A NFS client partial file corruption problem in recent/current kernels

A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

 We've found a readily reproducable situation where the current
NFS client code will provide zero bytes instead of actual data at the
end of the file (sort of) to user programs. This can result in program
failure, or permanent file corruption if the program reading the file
writes the bad data back to the file; otherwise, the corruption goes
away when the client's cached data is pushed out of memory (or explicitly
dropped by dropping the pagecache through /proc/sys/vm/drop_caches).

 The reproduction steps are:

* on a NFS client, open the file read-write and read to the end of the
  file (possibly just read the end of the file).
* hold the file open read-write and wait for the file size to grow.

  All the bits of these first two steps appear to be required; you must
  read the end of the file, you must have the file open read-write,
  and you must hold it open read-write.

* on either another NFS client or the NFS server, append data to the
  file.

* now that your program sees the new file size, try to read the new
  data (from the old end of the file to the new end of the file).
  Any data from the old end of file up to the next 4 KB boundary will
  be zero bytes instead of its actual content; after that, it will be
  the proper new content.

I have a demonstration reproduction program here:
	https://www.cs.toronto.edu/~cks/vendors/linux-nfs/

This issue isn't present in the Ubuntu 16.04 LTS server kernel (labeled
as '4.4.0', plus years of Ubuntu changes) and is present in the Ubuntu
18.04 LTS kernel (labeled 4.15.0) and the Fedora 28 4.17.9 and 4.18.5
kernels. It happens on both NFSv3 and NFSv4 mounts (both with 'sec=sys')
and the NFS fileserver OS and the filesystem type (on Linux) doesn't
appear to matter; we initially saw this against OmniOS NFS servers using
ZFS and have reproduced this against Linux NFS servers on ext4, tmpfs,
and ZFS (ZFS on Linux) with both Ubuntu 18.04 and Fedora 28 kernels.

 This bug causes Alpine to fail when accessing your /var/mail inbox
over NFS (and you get new mail delivered to it). There are probably other
programs affected, although hopefully not many programs hold files open
read-write while other programs are appending data.

 I'd be happy to answer any further questions, but we have limited
ability to try different kernels or kernel changes to see if they change
the situation (we don't run stock kernels on any machines; they're all
vendor-based ones).

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDExOjU5IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl
Og0KPiAgV2UndmUgZm91bmQgYSByZWFkaWx5IHJlcHJvZHVjYWJsZSBzaXR1YXRpb24gd2hlcmUg
dGhlIGN1cnJlbnQNCj4gTkZTIGNsaWVudCBjb2RlIHdpbGwgcHJvdmlkZSB6ZXJvIGJ5dGVzIGlu
c3RlYWQgb2YgYWN0dWFsIGRhdGEgYXQgdGhlDQo+IGVuZCBvZiB0aGUgZmlsZSAoc29ydCBvZikg
dG8gdXNlciBwcm9ncmFtcy4gVGhpcyBjYW4gcmVzdWx0IGluDQo+IHByb2dyYW0NCj4gZmFpbHVy
ZSwgb3IgcGVybWFuZW50IGZpbGUgY29ycnVwdGlvbiBpZiB0aGUgcHJvZ3JhbSByZWFkaW5nIHRo
ZSBmaWxlDQo+IHdyaXRlcyB0aGUgYmFkIGRhdGEgYmFjayB0byB0aGUgZmlsZTsgb3RoZXJ3aXNl
LCB0aGUgY29ycnVwdGlvbiBnb2VzDQo+IGF3YXkgd2hlbiB0aGUgY2xpZW50J3MgY2FjaGVkIGRh
dGEgaXMgcHVzaGVkIG91dCBvZiBtZW1vcnkgKG9yDQo+IGV4cGxpY2l0bHkNCj4gZHJvcHBlZCBi
eSBkcm9wcGluZyB0aGUgcGFnZWNhY2hlIHRocm91Z2ggL3Byb2Mvc3lzL3ZtL2Ryb3BfY2FjaGVz
KS4NCj4gDQo+ICBUaGUgcmVwcm9kdWN0aW9uIHN0ZXBzIGFyZToNCj4gDQo+ICogb24gYSBORlMg
Y2xpZW50LCBvcGVuIHRoZSBmaWxlIHJlYWQtd3JpdGUgYW5kIHJlYWQgdG8gdGhlIGVuZCBvZg0K
PiB0aGUNCj4gICBmaWxlIChwb3NzaWJseSBqdXN0IHJlYWQgdGhlIGVuZCBvZiB0aGUgZmlsZSku
DQo+ICogaG9sZCB0aGUgZmlsZSBvcGVuIHJlYWQtd3JpdGUgYW5kIHdhaXQgZm9yIHRoZSBmaWxl
IHNpemUgdG8gZ3Jvdy4NCj4gDQo+ICAgQWxsIHRoZSBiaXRzIG9mIHRoZXNlIGZpcnN0IHR3byBz
dGVwcyBhcHBlYXIgdG8gYmUgcmVxdWlyZWQ7IHlvdQ0KPiBtdXN0DQo+ICAgcmVhZCB0aGUgZW5k
IG9mIHRoZSBmaWxlLCB5b3UgbXVzdCBoYXZlIHRoZSBmaWxlIG9wZW4gcmVhZC13cml0ZSwNCj4g
ICBhbmQgeW91IG11c3QgaG9sZCBpdCBvcGVuIHJlYWQtd3JpdGUuDQo+IA0KPiAqIG9uIGVpdGhl
ciBhbm90aGVyIE5GUyBjbGllbnQgb3IgdGhlIE5GUyBzZXJ2ZXIsIGFwcGVuZCBkYXRhIHRvIHRo
ZQ0KPiAgIGZpbGUuDQo+IA0KPiAqIG5vdyB0aGF0IHlvdXIgcHJvZ3JhbSBzZWVzIHRoZSBuZXcg
ZmlsZSBzaXplLCB0cnkgdG8gcmVhZCB0aGUgbmV3DQo+ICAgZGF0YSAoZnJvbSB0aGUgb2xkIGVu
ZCBvZiB0aGUgZmlsZSB0byB0aGUgbmV3IGVuZCBvZiB0aGUgZmlsZSkuDQo+ICAgQW55IGRhdGEg
ZnJvbSB0aGUgb2xkIGVuZCBvZiBmaWxlIHVwIHRvIHRoZSBuZXh0IDQgS0IgYm91bmRhcnkgd2ls
bA0KPiAgIGJlIHplcm8gYnl0ZXMgaW5zdGVhZCBvZiBpdHMgYWN0dWFsIGNvbnRlbnQ7IGFmdGVy
IHRoYXQsIGl0IHdpbGwgYmUNCj4gICB0aGUgcHJvcGVyIG5ldyBjb250ZW50Lg0KPiANCj4gSSBo
YXZlIGEgZGVtb25zdHJhdGlvbiByZXByb2R1Y3Rpb24gcHJvZ3JhbSBoZXJlOg0KPiAJaHR0cHM6
Ly93d3cuY3MudG9yb250by5lZHUvfmNrcy92ZW5kb3JzL2xpbnV4LW5mcy8NCj4gDQo+IFRoaXMg
aXNzdWUgaXNuJ3QgcHJlc2VudCBpbiB0aGUgVWJ1bnR1IDE2LjA0IExUUyBzZXJ2ZXIga2VybmVs
DQo+IChsYWJlbGVkDQo+IGFzICc0LjQuMCcsIHBsdXMgeWVhcnMgb2YgVWJ1bnR1IGNoYW5nZXMp
IGFuZCBpcyBwcmVzZW50IGluIHRoZQ0KPiBVYnVudHUNCj4gMTguMDQgTFRTIGtlcm5lbCAobGFi
ZWxlZCA0LjE1LjApIGFuZCB0aGUgRmVkb3JhIDI4IDQuMTcuOSBhbmQgNC4xOC41DQo+IGtlcm5l
bHMuIEl0IGhhcHBlbnMgb24gYm90aCBORlN2MyBhbmQgTkZTdjQgbW91bnRzIChib3RoIHdpdGgN
Cj4gJ3NlYz1zeXMnKQ0KPiBhbmQgdGhlIE5GUyBmaWxlc2VydmVyIE9TIGFuZCB0aGUgZmlsZXN5
c3RlbSB0eXBlIChvbiBMaW51eCkgZG9lc24ndA0KPiBhcHBlYXIgdG8gbWF0dGVyOyB3ZSBpbml0
aWFsbHkgc2F3IHRoaXMgYWdhaW5zdCBPbW5pT1MgTkZTIHNlcnZlcnMNCj4gdXNpbmcNCj4gWkZT
IGFuZCBoYXZlIHJlcHJvZHVjZWQgdGhpcyBhZ2FpbnN0IExpbnV4IE5GUyBzZXJ2ZXJzIG9uIGV4
dDQsDQo+IHRtcGZzLA0KPiBhbmQgWkZTIChaRlMgb24gTGludXgpIHdpdGggYm90aCBVYnVudHUg
MTguMDQgYW5kIEZlZG9yYSAyOCBrZXJuZWxzLg0KPiANCj4gIFRoaXMgYnVnIGNhdXNlcyBBbHBp
bmUgdG8gZmFpbCB3aGVuIGFjY2Vzc2luZyB5b3VyIC92YXIvbWFpbCBpbmJveA0KPiBvdmVyIE5G
UyAoYW5kIHlvdSBnZXQgbmV3IG1haWwgZGVsaXZlcmVkIHRvIGl0KS4gVGhlcmUgYXJlIHByb2Jh
Ymx5DQo+IG90aGVyDQo+IHByb2dyYW1zIGFmZmVjdGVkLCBhbHRob3VnaCBob3BlZnVsbHkgbm90
IG1hbnkgcHJvZ3JhbXMgaG9sZCBmaWxlcw0KPiBvcGVuDQo+IHJlYWQtd3JpdGUgd2hpbGUgb3Ro
ZXIgcHJvZ3JhbXMgYXJlIGFwcGVuZGluZyBkYXRhLg0KPiANCj4gIEknZCBiZSBoYXBweSB0byBh
bnN3ZXIgYW55IGZ1cnRoZXIgcXVlc3Rpb25zLCBidXQgd2UgaGF2ZSBsaW1pdGVkDQo+IGFiaWxp
dHkgdG8gdHJ5IGRpZmZlcmVudCBrZXJuZWxzIG9yIGtlcm5lbCBjaGFuZ2VzIHRvIHNlZSBpZiB0
aGV5DQo+IGNoYW5nZQ0KPiB0aGUgc2l0dWF0aW9uICh3ZSBkb24ndCBydW4gc3RvY2sga2VybmVs
cyBvbiBhbnkgbWFjaGluZXM7IHRoZXkncmUNCj4gYWxsDQo+IHZlbmRvci1iYXNlZCBvbmVzKS4N
Cj4gDQoNClBsZWFzZSBzZWUgaHR0cDovL25mcy5zb3VyY2Vmb3JnZS5uZXQvI2ZhcV9hOA0KDQoN
Ci0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXIsIEhhbW1l
cnNwYWNlDQp0cm9uZC5teWtsZWJ1c3RAaGFtbWVyc3BhY2UuY29tDQoNCg0K

Re: A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

> >  We've found a readily reproducable situation where the current
> > NFS client code will provide zero bytes instead of actual data at
> > the end of the file (sort of) to user programs. This can result
> > in program failure, or permanent file corruption if the program
> > reading the file writes the bad data back to the file; otherwise,
> > the corruption goes away when the client's cached data is pushed out
> > of memory (or explicitly dropped by dropping the pagecache through
> > /proc/sys/vm/drop_caches).
[...]
> Please see http://nfs.sourceforge.net/#faq_a8

 I don't think this is a close to open consistency issue, or if it is
I would argue that it is a clear bug on the Linux NFS client. I have
a number of reasons for saying this:

- the client clearly sees the new attributes; it knows that the file
  has been extended from the previous state that it knew of. My demo
  program specifically waits until user-level fstat() returns a different
  result, which I believe means that the client kernel has seen a different
  GETATTR result and so should have purged its cache (based on what the
  FAQ says).

  (Unless the FAQ means that the kernel absolutely refuses to guarantee
  anything about file consistency unless you close and then reopen the
  file, even if it *knows* that the file has changed on the server,
  which isn't clear from how the FAQ is currently written.)

- the client is fetching some new data from the fileserver (data after
  the partial 4 KB page at the old end of the file).

- the client isn't writing to the file in my demonstration program; it's
  only opening it in read-write mode and then reading it. Also, this
  doesn't happen if the client does exactly the same set of operations
  but has the file open read-only (with it staying open throughout).

- this didn't happen in older kernels.

In addition, although I didn't mention it in my original email, this
happens on a NFS filesystem mounted 'noac'.

Pragmatically, Alpine used to work with NFS mounted filesystems where
email was appended to them from other machines and it no longer does,
and the only difference is the kernel version involved on the client.
This breakage is actively dangerous.

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDE0OjAyIC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl
Og0KPiA+ID4gIFdlJ3ZlIGZvdW5kIGEgcmVhZGlseSByZXByb2R1Y2FibGUgc2l0dWF0aW9uIHdo
ZXJlIHRoZSBjdXJyZW50DQo+ID4gPiBORlMgY2xpZW50IGNvZGUgd2lsbCBwcm92aWRlIHplcm8g
Ynl0ZXMgaW5zdGVhZCBvZiBhY3R1YWwgZGF0YSBhdA0KPiA+ID4gdGhlIGVuZCBvZiB0aGUgZmls
ZSAoc29ydCBvZikgdG8gdXNlciBwcm9ncmFtcy4gVGhpcyBjYW4gcmVzdWx0DQo+ID4gPiBpbiBw
cm9ncmFtIGZhaWx1cmUsIG9yIHBlcm1hbmVudCBmaWxlIGNvcnJ1cHRpb24gaWYgdGhlIHByb2dy
YW0NCj4gPiA+IHJlYWRpbmcgdGhlIGZpbGUgd3JpdGVzIHRoZSBiYWQgZGF0YSBiYWNrIHRvIHRo
ZSBmaWxlOyBvdGhlcndpc2UsDQo+ID4gPiB0aGUgY29ycnVwdGlvbiBnb2VzIGF3YXkgd2hlbiB0
aGUgY2xpZW50J3MgY2FjaGVkIGRhdGEgaXMgcHVzaGVkDQo+ID4gPiBvdXQNCj4gPiA+IG9mIG1l
bW9yeSAob3IgZXhwbGljaXRseSBkcm9wcGVkIGJ5IGRyb3BwaW5nIHRoZSBwYWdlY2FjaGUNCj4g
PiA+IHRocm91Z2gNCj4gPiA+IC9wcm9jL3N5cy92bS9kcm9wX2NhY2hlcykuDQo+IA0KPiBbLi4u
XQ0KPiA+IFBsZWFzZSBzZWUgaHR0cDovL25mcy5zb3VyY2Vmb3JnZS5uZXQvI2ZhcV9hOA0KPiAN
Cj4gIEkgZG9uJ3QgdGhpbmsgdGhpcyBpcyBhIGNsb3NlIHRvIG9wZW4gY29uc2lzdGVuY3kgaXNz
dWUsIG9yIGlmIGl0IGlzDQo+IEkgd291bGQgYXJndWUgdGhhdCBpdCBpcyBhIGNsZWFyIGJ1ZyBv
biB0aGUgTGludXggTkZTIGNsaWVudC4gSSBoYXZlDQo+IGEgbnVtYmVyIG9mIHJlYXNvbnMgZm9y
IHNheWluZyB0aGlzOg0KPiANCj4gLSB0aGUgY2xpZW50IGNsZWFybHkgc2VlcyB0aGUgbmV3IGF0
dHJpYnV0ZXM7IGl0IGtub3dzIHRoYXQgdGhlIGZpbGUNCj4gICBoYXMgYmVlbiBleHRlbmRlZCBm
cm9tIHRoZSBwcmV2aW91cyBzdGF0ZSB0aGF0IGl0IGtuZXcgb2YuIE15IGRlbW8NCj4gICBwcm9n
cmFtIHNwZWNpZmljYWxseSB3YWl0cyB1bnRpbCB1c2VyLWxldmVsIGZzdGF0KCkgcmV0dXJucyBh
DQo+IGRpZmZlcmVudA0KPiAgIHJlc3VsdCwgd2hpY2ggSSBiZWxpZXZlIG1lYW5zIHRoYXQgdGhl
IGNsaWVudCBrZXJuZWwgaGFzIHNlZW4gYQ0KPiBkaWZmZXJlbnQNCj4gICBHRVRBVFRSIHJlc3Vs
dCBhbmQgc28gc2hvdWxkIGhhdmUgcHVyZ2VkIGl0cyBjYWNoZSAoYmFzZWQgb24gd2hhdA0KPiB0
aGUNCj4gICBGQVEgc2F5cykuDQo+IA0KPiAgIChVbmxlc3MgdGhlIEZBUSBtZWFucyB0aGF0IHRo
ZSBrZXJuZWwgYWJzb2x1dGVseSByZWZ1c2VzIHRvDQo+IGd1YXJhbnRlZQ0KPiAgIGFueXRoaW5n
IGFib3V0IGZpbGUgY29uc2lzdGVuY3kgdW5sZXNzIHlvdSBjbG9zZSBhbmQgdGhlbiByZW9wZW4N
Cj4gdGhlDQo+ICAgZmlsZSwgZXZlbiBpZiBpdCAqa25vd3MqIHRoYXQgdGhlIGZpbGUgaGFzIGNo
YW5nZWQgb24gdGhlIHNlcnZlciwNCj4gICB3aGljaCBpc24ndCBjbGVhciBmcm9tIGhvdyB0aGUg
RkFRIGlzIGN1cnJlbnRseSB3cml0dGVuLikNCj4gDQo+IC0gdGhlIGNsaWVudCBpcyBmZXRjaGlu
ZyBzb21lIG5ldyBkYXRhIGZyb20gdGhlIGZpbGVzZXJ2ZXIgKGRhdGENCj4gYWZ0ZXINCj4gICB0
aGUgcGFydGlhbCA0IEtCIHBhZ2UgYXQgdGhlIG9sZCBlbmQgb2YgdGhlIGZpbGUpLg0KPiANCj4g
LSB0aGUgY2xpZW50IGlzbid0IHdyaXRpbmcgdG8gdGhlIGZpbGUgaW4gbXkgZGVtb25zdHJhdGlv
biBwcm9ncmFtOw0KPiBpdCdzDQo+ICAgb25seSBvcGVuaW5nIGl0IGluIHJlYWQtd3JpdGUgbW9k
ZSBhbmQgdGhlbiByZWFkaW5nIGl0LiBBbHNvLCB0aGlzDQo+ICAgZG9lc24ndCBoYXBwZW4gaWYg
dGhlIGNsaWVudCBkb2VzIGV4YWN0bHkgdGhlIHNhbWUgc2V0IG9mDQo+IG9wZXJhdGlvbnMNCj4g
ICBidXQgaGFzIHRoZSBmaWxlIG9wZW4gcmVhZC1vbmx5ICh3aXRoIGl0IHN0YXlpbmcgb3BlbiB0
aHJvdWdob3V0KS4NCj4gDQo+IC0gdGhpcyBkaWRuJ3QgaGFwcGVuIGluIG9sZGVyIGtlcm5lbHMu
DQo+IA0KPiBJbiBhZGRpdGlvbiwgYWx0aG91Z2ggSSBkaWRuJ3QgbWVudGlvbiBpdCBpbiBteSBv
cmlnaW5hbCBlbWFpbCwgdGhpcw0KPiBoYXBwZW5zIG9uIGEgTkZTIGZpbGVzeXN0ZW0gbW91bnRl
ZCAnbm9hYycuDQo+IA0KPiBQcmFnbWF0aWNhbGx5LCBBbHBpbmUgdXNlZCB0byB3b3JrIHdpdGgg
TkZTIG1vdW50ZWQgZmlsZXN5c3RlbXMgd2hlcmUNCj4gZW1haWwgd2FzIGFwcGVuZGVkIHRvIHRo
ZW0gZnJvbSBvdGhlciBtYWNoaW5lcyBhbmQgaXQgbm8gbG9uZ2VyIGRvZXMsDQo+IGFuZCB0aGUg
b25seSBkaWZmZXJlbmNlIGlzIHRoZSBrZXJuZWwgdmVyc2lvbiBpbnZvbHZlZCBvbiB0aGUgY2xp
ZW50Lg0KPiBUaGlzIGJyZWFrYWdlIGlzIGFjdGl2ZWx5IGRhbmdlcm91cy4NCg0KU3VyZSwgYnV0
IHVubGVzcyB5b3UgYXJlIGxvY2tpbmcgdGhlIGZpbGUsIG9yIHlvdSBhcmUgZXhwbGljaXRseSB1
c2luZw0KT19ESVJFQ1QgdG8gZG8gdW5jYWNoZWQgSS9PLCB0aGVuIHlvdSBhcmUgaW4gdmlvbGF0
aW9uIG9mIHRoZSBjbG9zZS10by0NCm9wZW4gY29uc2lzdGVuY3kgbW9kZWwsIGFuZCB0aGUgY2xp
ZW50IGlzIGdvaW5nIHRvIGJlaGF2ZSBhcyB5b3UNCmRlc2NyaWJlIGFib3ZlLiBORlMgdXNlcyBh
IGRpc3RyaWJ1dGVkIGZpbGVzeXN0ZW0gbW9kZWwsIG5vdCBhDQpjbHVzdGVyZWQgb25lLg0KDQot
LSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyLCBIYW1tZXJz
cGFjZQ0KdHJvbmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNvbQ0KDQoNCg==

Re: A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

> > Pragmatically, Alpine used to work with NFS mounted filesystems where
> > email was appended to them from other machines and it no longer does,
> > and the only difference is the kernel version involved on the client.
> > This breakage is actively dangerous.
> 
> Sure, but unless you are locking the file, or you are explicitly using
> O_DIRECT to do uncached I/O, then you are in violation of the close-to-
> open consistency model, and the client is going to behave as you
> describe above. NFS uses a distributed filesystem model, not a
> clustered one.

 In the close to open consistency model, is it legal and proper to
do the following sequence:

- open a file read-write
- fstat() the file until the reported file size changes
- close the file; open it again read-write
- read new data from the file

If this sequence is legal, then I think there is a bug, because I can
make the zero bytes appear even with this sequence. I've updated my
reproduction program, in

	https://www.cs.toronto.edu/~cks/vendors/linux-nfs/

to have a '--reopen' option that does this.

If this sequence is not legal and can legally result in corrupted
data in the file, then I think there is a potential problem, because
it creates a situation where one program (opening the file read-write
and holding it open) could cause corruption for another program (which
properly opens and closes the file). I can reproduce this with two
running instances of my test program. Perhaps this is considered
invalid because it is a violation of close to open across the entire
client kernel, but if so I feel this is dangerous; it puts all programs
reading NFS mounted files at the mercy of everything else on the
system, no matter how much they try to do the right thing. They can
open it read only and close it while they wait for changes and then
reopen it read only afterward, and they will still get corrupted data.

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Chuck Lever]

> On Sep 11, 2018, at 4:00 PM, Trond Myklebust <trondmy@hammerspace.com> =
wrote:
>=20
> On Tue, 2018-09-11 at 14:02 -0400, Chris Siebenmann wrote:
>>>> We've found a readily reproducable situation where the current
>>>> NFS client code will provide zero bytes instead of actual data at
>>>> the end of the file (sort of) to user programs. This can result
>>>> in program failure, or permanent file corruption if the program
>>>> reading the file writes the bad data back to the file; otherwise,
>>>> the corruption goes away when the client's cached data is pushed
>>>> out
>>>> of memory (or explicitly dropped by dropping the pagecache
>>>> through
>>>> /proc/sys/vm/drop_caches).
>>=20
>> [...]
>>> Please see http://nfs.sourceforge.net/#faq_a8
>>=20
>> I don't think this is a close to open consistency issue, or if it is
>> I would argue that it is a clear bug on the Linux NFS client. I have
>> a number of reasons for saying this:
>>=20
>> - the client clearly sees the new attributes; it knows that the file
>>  has been extended from the previous state that it knew of. My demo
>>  program specifically waits until user-level fstat() returns a
>> different
>>  result, which I believe means that the client kernel has seen a
>> different
>>  GETATTR result and so should have purged its cache (based on what
>> the
>>  FAQ says).
>>=20
>>  (Unless the FAQ means that the kernel absolutely refuses to
>> guarantee
>>  anything about file consistency unless you close and then reopen
>> the
>>  file, even if it *knows* that the file has changed on the server,
>>  which isn't clear from how the FAQ is currently written.)
>>=20
>> - the client is fetching some new data from the fileserver (data
>> after
>>  the partial 4 KB page at the old end of the file).
>>=20
>> - the client isn't writing to the file in my demonstration program;
>> it's
>>  only opening it in read-write mode and then reading it. Also, this
>>  doesn't happen if the client does exactly the same set of
>> operations
>>  but has the file open read-only (with it staying open throughout).
>>=20
>> - this didn't happen in older kernels.
>>=20
>> In addition, although I didn't mention it in my original email, this
>> happens on a NFS filesystem mounted 'noac'.
>>=20
>> Pragmatically, Alpine used to work with NFS mounted filesystems where
>> email was appended to them from other machines and it no longer does,
>> and the only difference is the kernel version involved on the client.
>> This breakage is actively dangerous.
>=20
> Sure, but unless you are locking the file, or you are explicitly using
> O_DIRECT to do uncached I/O, then you are in violation of the =
close-to-
> open consistency model, and the client is going to behave as you
> describe above. NFS uses a distributed filesystem model, not a
> clustered one.

I would expect Alpine to work if "vers=3D3,noac" is in use.


--
Chuck Lever

Re: A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

> >> Pragmatically, Alpine used to work with NFS mounted filesystems where
> >> email was appended to them from other machines and it no longer does,
> >> and the only difference is the kernel version involved on the client.
> >> This breakage is actively dangerous.
> > 
> > Sure, but unless you are locking the file, or you are explicitly using
> > O_DIRECT to do uncached I/O, then you are in violation of the close-to-
> > open consistency model, and the client is going to behave as you
> > describe above. NFS uses a distributed filesystem model, not a
> > clustered one.
> 
> I would expect Alpine to work if "vers=3,noac" is in use.

 We're setting noac (and vers=3) for our /var/mail NFS mount, but
perhaps we've got some additional option that is wrong here and
should be changed? According to /proc/mounts, our mount settings
for /var/mail are:

	fs0.cs.toronto.edu:/cs/mail /var/mail nfs rw,sync,nosuid,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,noacl,proto=tcp,timeo=11,retrans=4,sec=sys,mountaddr=128.100.3.130,mountvers=3,mountport=34630,mountproto=udp,local_lock=none,addr=128.100.3.130

(To an OmniOS fileserver with ZFS.)

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDE2OjQwIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
PiBPbiBTZXAgMTEsIDIwMTgsIGF0IDQ6MDAgUE0sIFRyb25kIE15a2xlYnVzdCA8DQo+ID4gdHJv
bmRteUBoYW1tZXJzcGFjZS5jb20+IHdyb3RlOg0KPiA+IA0KPiA+IE9uIFR1ZSwgMjAxOC0wOS0x
MSBhdCAxNDowMiAtMDQwMCwgQ2hyaXMgU2llYmVubWFubiB3cm90ZToNCj4gPiA+ID4gPiBXZSd2
ZSBmb3VuZCBhIHJlYWRpbHkgcmVwcm9kdWNhYmxlIHNpdHVhdGlvbiB3aGVyZSB0aGUNCj4gPiA+
ID4gPiBjdXJyZW50DQo+ID4gPiA+ID4gTkZTIGNsaWVudCBjb2RlIHdpbGwgcHJvdmlkZSB6ZXJv
IGJ5dGVzIGluc3RlYWQgb2YgYWN0dWFsDQo+ID4gPiA+ID4gZGF0YSBhdA0KPiA+ID4gPiA+IHRo
ZSBlbmQgb2YgdGhlIGZpbGUgKHNvcnQgb2YpIHRvIHVzZXIgcHJvZ3JhbXMuIFRoaXMgY2FuDQo+
ID4gPiA+ID4gcmVzdWx0DQo+ID4gPiA+ID4gaW4gcHJvZ3JhbSBmYWlsdXJlLCBvciBwZXJtYW5l
bnQgZmlsZSBjb3JydXB0aW9uIGlmIHRoZQ0KPiA+ID4gPiA+IHByb2dyYW0NCj4gPiA+ID4gPiBy
ZWFkaW5nIHRoZSBmaWxlIHdyaXRlcyB0aGUgYmFkIGRhdGEgYmFjayB0byB0aGUgZmlsZTsNCj4g
PiA+ID4gPiBvdGhlcndpc2UsDQo+ID4gPiA+ID4gdGhlIGNvcnJ1cHRpb24gZ29lcyBhd2F5IHdo
ZW4gdGhlIGNsaWVudCdzIGNhY2hlZCBkYXRhIGlzDQo+ID4gPiA+ID4gcHVzaGVkDQo+ID4gPiA+
ID4gb3V0DQo+ID4gPiA+ID4gb2YgbWVtb3J5IChvciBleHBsaWNpdGx5IGRyb3BwZWQgYnkgZHJv
cHBpbmcgdGhlIHBhZ2VjYWNoZQ0KPiA+ID4gPiA+IHRocm91Z2gNCj4gPiA+ID4gPiAvcHJvYy9z
eXMvdm0vZHJvcF9jYWNoZXMpLg0KPiA+ID4gDQo+ID4gPiBbLi4uXQ0KPiA+ID4gPiBQbGVhc2Ug
c2VlIGh0dHA6Ly9uZnMuc291cmNlZm9yZ2UubmV0LyNmYXFfYTgNCj4gPiA+IA0KPiA+ID4gSSBk
b24ndCB0aGluayB0aGlzIGlzIGEgY2xvc2UgdG8gb3BlbiBjb25zaXN0ZW5jeSBpc3N1ZSwgb3Ig
aWYgaXQNCj4gPiA+IGlzDQo+ID4gPiBJIHdvdWxkIGFyZ3VlIHRoYXQgaXQgaXMgYSBjbGVhciBi
dWcgb24gdGhlIExpbnV4IE5GUyBjbGllbnQuIEkNCj4gPiA+IGhhdmUNCj4gPiA+IGEgbnVtYmVy
IG9mIHJlYXNvbnMgZm9yIHNheWluZyB0aGlzOg0KPiA+ID4gDQo+ID4gPiAtIHRoZSBjbGllbnQg
Y2xlYXJseSBzZWVzIHRoZSBuZXcgYXR0cmlidXRlczsgaXQga25vd3MgdGhhdCB0aGUNCj4gPiA+
IGZpbGUNCj4gPiA+ICBoYXMgYmVlbiBleHRlbmRlZCBmcm9tIHRoZSBwcmV2aW91cyBzdGF0ZSB0
aGF0IGl0IGtuZXcgb2YuIE15DQo+ID4gPiBkZW1vDQo+ID4gPiAgcHJvZ3JhbSBzcGVjaWZpY2Fs
bHkgd2FpdHMgdW50aWwgdXNlci1sZXZlbCBmc3RhdCgpIHJldHVybnMgYQ0KPiA+ID4gZGlmZmVy
ZW50DQo+ID4gPiAgcmVzdWx0LCB3aGljaCBJIGJlbGlldmUgbWVhbnMgdGhhdCB0aGUgY2xpZW50
IGtlcm5lbCBoYXMgc2VlbiBhDQo+ID4gPiBkaWZmZXJlbnQNCj4gPiA+ICBHRVRBVFRSIHJlc3Vs
dCBhbmQgc28gc2hvdWxkIGhhdmUgcHVyZ2VkIGl0cyBjYWNoZSAoYmFzZWQgb24NCj4gPiA+IHdo
YXQNCj4gPiA+IHRoZQ0KPiA+ID4gIEZBUSBzYXlzKS4NCj4gPiA+IA0KPiA+ID4gIChVbmxlc3Mg
dGhlIEZBUSBtZWFucyB0aGF0IHRoZSBrZXJuZWwgYWJzb2x1dGVseSByZWZ1c2VzIHRvDQo+ID4g
PiBndWFyYW50ZWUNCj4gPiA+ICBhbnl0aGluZyBhYm91dCBmaWxlIGNvbnNpc3RlbmN5IHVubGVz
cyB5b3UgY2xvc2UgYW5kIHRoZW4gcmVvcGVuDQo+ID4gPiB0aGUNCj4gPiA+ICBmaWxlLCBldmVu
IGlmIGl0ICprbm93cyogdGhhdCB0aGUgZmlsZSBoYXMgY2hhbmdlZCBvbiB0aGUNCj4gPiA+IHNl
cnZlciwNCj4gPiA+ICB3aGljaCBpc24ndCBjbGVhciBmcm9tIGhvdyB0aGUgRkFRIGlzIGN1cnJl
bnRseSB3cml0dGVuLikNCj4gPiA+IA0KPiA+ID4gLSB0aGUgY2xpZW50IGlzIGZldGNoaW5nIHNv
bWUgbmV3IGRhdGEgZnJvbSB0aGUgZmlsZXNlcnZlciAoZGF0YQ0KPiA+ID4gYWZ0ZXINCj4gPiA+
ICB0aGUgcGFydGlhbCA0IEtCIHBhZ2UgYXQgdGhlIG9sZCBlbmQgb2YgdGhlIGZpbGUpLg0KPiA+
ID4gDQo+ID4gPiAtIHRoZSBjbGllbnQgaXNuJ3Qgd3JpdGluZyB0byB0aGUgZmlsZSBpbiBteSBk
ZW1vbnN0cmF0aW9uDQo+ID4gPiBwcm9ncmFtOw0KPiA+ID4gaXQncw0KPiA+ID4gIG9ubHkgb3Bl
bmluZyBpdCBpbiByZWFkLXdyaXRlIG1vZGUgYW5kIHRoZW4gcmVhZGluZyBpdC4gQWxzbywNCj4g
PiA+IHRoaXMNCj4gPiA+ICBkb2Vzbid0IGhhcHBlbiBpZiB0aGUgY2xpZW50IGRvZXMgZXhhY3Rs
eSB0aGUgc2FtZSBzZXQgb2YNCj4gPiA+IG9wZXJhdGlvbnMNCj4gPiA+ICBidXQgaGFzIHRoZSBm
aWxlIG9wZW4gcmVhZC1vbmx5ICh3aXRoIGl0IHN0YXlpbmcgb3Blbg0KPiA+ID4gdGhyb3VnaG91
dCkuDQo+ID4gPiANCj4gPiA+IC0gdGhpcyBkaWRuJ3QgaGFwcGVuIGluIG9sZGVyIGtlcm5lbHMu
DQo+ID4gPiANCj4gPiA+IEluIGFkZGl0aW9uLCBhbHRob3VnaCBJIGRpZG4ndCBtZW50aW9uIGl0
IGluIG15IG9yaWdpbmFsIGVtYWlsLA0KPiA+ID4gdGhpcw0KPiA+ID4gaGFwcGVucyBvbiBhIE5G
UyBmaWxlc3lzdGVtIG1vdW50ZWQgJ25vYWMnLg0KPiA+ID4gDQo+ID4gPiBQcmFnbWF0aWNhbGx5
LCBBbHBpbmUgdXNlZCB0byB3b3JrIHdpdGggTkZTIG1vdW50ZWQgZmlsZXN5c3RlbXMNCj4gPiA+
IHdoZXJlDQo+ID4gPiBlbWFpbCB3YXMgYXBwZW5kZWQgdG8gdGhlbSBmcm9tIG90aGVyIG1hY2hp
bmVzIGFuZCBpdCBubyBsb25nZXINCj4gPiA+IGRvZXMsDQo+ID4gPiBhbmQgdGhlIG9ubHkgZGlm
ZmVyZW5jZSBpcyB0aGUga2VybmVsIHZlcnNpb24gaW52b2x2ZWQgb24gdGhlDQo+ID4gPiBjbGll
bnQuDQo+ID4gPiBUaGlzIGJyZWFrYWdlIGlzIGFjdGl2ZWx5IGRhbmdlcm91cy4NCj4gPiANCj4g
PiBTdXJlLCBidXQgdW5sZXNzIHlvdSBhcmUgbG9ja2luZyB0aGUgZmlsZSwgb3IgeW91IGFyZSBl
eHBsaWNpdGx5DQo+ID4gdXNpbmcNCj4gPiBPX0RJUkVDVCB0byBkbyB1bmNhY2hlZCBJL08sIHRo
ZW4geW91IGFyZSBpbiB2aW9sYXRpb24gb2YgdGhlDQo+ID4gY2xvc2UtdG8tDQo+ID4gb3BlbiBj
b25zaXN0ZW5jeSBtb2RlbCwgYW5kIHRoZSBjbGllbnQgaXMgZ29pbmcgdG8gYmVoYXZlIGFzIHlv
dQ0KPiA+IGRlc2NyaWJlIGFib3ZlLiBORlMgdXNlcyBhIGRpc3RyaWJ1dGVkIGZpbGVzeXN0ZW0g
bW9kZWwsIG5vdCBhDQo+ID4gY2x1c3RlcmVkIG9uZS4NCj4gDQo+IEkgd291bGQgZXhwZWN0IEFs
cGluZSB0byB3b3JrIGlmICJ2ZXJzPTMsbm9hYyIgaXMgaW4gdXNlLg0KPiANCg0Kbm9hYyBoYXMg
bm90aGluZyBhdCBhbGwgdG8gZG8gd2l0aCBkYXRhIGNhY2hlIGNvbnNpc3RlbmN5Lg0KDQotLSAN
ClRyb25kIE15a2xlYnVzdA0KQ1RPLCBIYW1tZXJzcGFjZSBJbmMNCjQzMDAgRWwgQ2FtaW5vIFJl
YWwsIFN1aXRlIDEwNQ0KTG9zIEFsdG9zLCBDQSA5NDAyMg0Kd3d3LmhhbW1lci5zcGFjZQ0KDQoN
Cg==

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDE2OjQ3IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl
Og0KPiA+ID4gPiBQcmFnbWF0aWNhbGx5LCBBbHBpbmUgdXNlZCB0byB3b3JrIHdpdGggTkZTIG1v
dW50ZWQgZmlsZXN5c3RlbXMNCj4gPiA+ID4gd2hlcmUNCj4gPiA+ID4gZW1haWwgd2FzIGFwcGVu
ZGVkIHRvIHRoZW0gZnJvbSBvdGhlciBtYWNoaW5lcyBhbmQgaXQgbm8gbG9uZ2VyDQo+ID4gPiA+
IGRvZXMsDQo+ID4gPiA+IGFuZCB0aGUgb25seSBkaWZmZXJlbmNlIGlzIHRoZSBrZXJuZWwgdmVy
c2lvbiBpbnZvbHZlZCBvbiB0aGUNCj4gPiA+ID4gY2xpZW50Lg0KPiA+ID4gPiBUaGlzIGJyZWFr
YWdlIGlzIGFjdGl2ZWx5IGRhbmdlcm91cy4NCj4gPiA+IA0KPiA+ID4gU3VyZSwgYnV0IHVubGVz
cyB5b3UgYXJlIGxvY2tpbmcgdGhlIGZpbGUsIG9yIHlvdSBhcmUgZXhwbGljaXRseQ0KPiA+ID4g
dXNpbmcNCj4gPiA+IE9fRElSRUNUIHRvIGRvIHVuY2FjaGVkIEkvTywgdGhlbiB5b3UgYXJlIGlu
IHZpb2xhdGlvbiBvZiB0aGUNCj4gPiA+IGNsb3NlLXRvLQ0KPiA+ID4gb3BlbiBjb25zaXN0ZW5j
eSBtb2RlbCwgYW5kIHRoZSBjbGllbnQgaXMgZ29pbmcgdG8gYmVoYXZlIGFzIHlvdQ0KPiA+ID4g
ZGVzY3JpYmUgYWJvdmUuIE5GUyB1c2VzIGEgZGlzdHJpYnV0ZWQgZmlsZXN5c3RlbSBtb2RlbCwg
bm90IGENCj4gPiA+IGNsdXN0ZXJlZCBvbmUuDQo+ID4gDQo+ID4gSSB3b3VsZCBleHBlY3QgQWxw
aW5lIHRvIHdvcmsgaWYgInZlcnM9Myxub2FjIiBpcyBpbiB1c2UuDQo+IA0KPiAgV2UncmUgc2V0
dGluZyBub2FjIChhbmQgdmVycz0zKSBmb3Igb3VyIC92YXIvbWFpbCBORlMgbW91bnQsIGJ1dA0K
PiBwZXJoYXBzIHdlJ3ZlIGdvdCBzb21lIGFkZGl0aW9uYWwgb3B0aW9uIHRoYXQgaXMgd3Jvbmcg
aGVyZSBhbmQNCj4gc2hvdWxkIGJlIGNoYW5nZWQ/IEFjY29yZGluZyB0byAvcHJvYy9tb3VudHMs
IG91ciBtb3VudCBzZXR0aW5ncw0KPiBmb3IgL3Zhci9tYWlsIGFyZToNCj4gDQo+IAlmczAuY3Mu
dG9yb250by5lZHU6L2NzL21haWwgL3Zhci9tYWlsIG5mcw0KPiBydyxzeW5jLG5vc3VpZCxub2Rl
dixyZWxhdGltZSx2ZXJzPTMscnNpemU9MTA0ODU3Nix3c2l6ZT0xMDQ4NTc2LG5hbWwNCj4gZW49
MjU1LGFjcmVnbWluPTAsYWNyZWdtYXg9MCxhY2Rpcm1pbj0wLGFjZGlybWF4PTAsaGFyZCxub2Fj
LG5vYWNsLHByDQo+IG90bz10Y3AsdGltZW89MTEscmV0cmFucz00LHNlYz1zeXMsbW91bnRhZGRy
PTEyOC4xMDAuMy4xMzAsbW91bnR2ZXJzPQ0KPiAzLG1vdW50cG9ydD0zNDYzMCxtb3VudHByb3Rv
PXVkcCxsb2NhbF9sb2NrPW5vbmUsYWRkcj0xMjguMTAwLjMuMTMwDQo+IA0KDQpUaGlzIGhhcyBu
b3RoaW5nIHRvIGRvIHdpdGggbW91bnQgb3B0aW9ucy4gQnVmZmVyZWQgcmVhZHMgb2YgYSBmaWxl
DQp0aGF0IGlzIGJlaW5nIHdyaXR0ZW4gdG8gb3ZlciBORlMgd2l0aG91dCB1c2luZyBsb2NraW5n
IGlzIGluaGVyZW50bHkNCnVuc2FmZS4gVGhhdCBhbHdheXMgaGFzIGJlZW4gdGhlIGNhc2UuLi4N
Cg0KQm90aCB3cml0ZXMgYW5kIHJlYWRzIGNhbiBiZSByZW9yZGVyZWQgYnkgdGhlIFJQQyBsYXll
ciBvbiBib3RoIHRoZQ0KY2xpZW50IGFuZCB0aGUgc2VydmVyLCBhbmQgdGhleSBjYW4gYmUgZnVy
dGhlciByZW9yZGVyZWQgYnkgdGhlIE5GUw0KbGF5ZXIgb24gdGhlIHNlcnZlci4gSW4gcHJhY3Rp
Y2UsIHRoaXMgbWVhbnMgdGhhdCB5b3UgY2FuIGZpbmQgeW91cnNlbGYNCnJlYWRpbmcgcGFydHMg
b2YgdGhlIGZpbGUgdGhhdCBoYXZlIG5vdCB5ZXQgY29tcGxldGVkIGJlaW5nIHdyaXR0ZW4gdG8s
DQpiZWNhdXNlLCBmb3IgZXhhbXBsZSwgYSB3cml0ZSB0aGF0IGV4dGVuZGVkIHRoZSBmaWxlIGZy
b20gb2Zmc2V0IDQwOTYtDQo4MTkxIGNvbXBsZXRlZCBiZWZvcmUgdGhlIHdyaXRlIHRoYXQgd2Fz
IHN1cHBvc2VkIHRvIGV4dGVuZCBpdCBmcm9tDQpvZmZzZXQgMC00MDk1IHdhcyBwcm9jZXNzZWQg
YnkgdGhlIHNlcnZlci4NCg0KVGhpcyBpcyB0cnVlIHdpdGggb3Igd2l0aG91dCBub2FjLi4uDQoN
Ci0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXIsIEhhbW1l
cnNwYWNlDQp0cm9uZC5teWtsZWJ1c3RAaGFtbWVyc3BhY2UuY29tDQoNCg0K

Re: A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

> This has nothing to do with mount options. Buffered reads of a file
> that is being written to over NFS without using locking is inherently
> unsafe. That always has been the case...
>
> Both writes and reads can be reordered by the RPC layer on both the
> client and the server, and they can be further reordered by the
> NFS layer on the server. In practice, this means that you can find
> yourself reading parts of the file that have not yet completed being
> written to, because, for example, a write that extended the file from
> offset 4096- 8191 completed before the write that was supposed to
> extend it from offset 0-4095 was processed by the server.

 Our issue also happens when the writes are done on the fileserver,
though, and they occur even if you allow plenty of time for the writes
to settle. I can run my test program in a mode where it explicitly waits
for me to tell it to continue, do the appending to the file on the
fileserver, 'sync' on the fileserver, wait five minutes, and the NFS
client will still see those zero bytes when it tries to read the new
data.

(To make sure the 'five minutes' bit wasn't hyperbole, I just tested
it.)

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDE3OjM5IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl
Og0KPiA+IFRoaXMgaGFzIG5vdGhpbmcgdG8gZG8gd2l0aCBtb3VudCBvcHRpb25zLiBCdWZmZXJl
ZCByZWFkcyBvZiBhIGZpbGUNCj4gPiB0aGF0IGlzIGJlaW5nIHdyaXR0ZW4gdG8gb3ZlciBORlMg
d2l0aG91dCB1c2luZyBsb2NraW5nIGlzDQo+ID4gaW5oZXJlbnRseQ0KPiA+IHVuc2FmZS4gVGhh
dCBhbHdheXMgaGFzIGJlZW4gdGhlIGNhc2UuLi4NCj4gPiANCj4gPiBCb3RoIHdyaXRlcyBhbmQg
cmVhZHMgY2FuIGJlIHJlb3JkZXJlZCBieSB0aGUgUlBDIGxheWVyIG9uIGJvdGggdGhlDQo+ID4g
Y2xpZW50IGFuZCB0aGUgc2VydmVyLCBhbmQgdGhleSBjYW4gYmUgZnVydGhlciByZW9yZGVyZWQg
YnkgdGhlDQo+ID4gTkZTIGxheWVyIG9uIHRoZSBzZXJ2ZXIuIEluIHByYWN0aWNlLCB0aGlzIG1l
YW5zIHRoYXQgeW91IGNhbiBmaW5kDQo+ID4geW91cnNlbGYgcmVhZGluZyBwYXJ0cyBvZiB0aGUg
ZmlsZSB0aGF0IGhhdmUgbm90IHlldCBjb21wbGV0ZWQNCj4gPiBiZWluZw0KPiA+IHdyaXR0ZW4g
dG8sIGJlY2F1c2UsIGZvciBleGFtcGxlLCBhIHdyaXRlIHRoYXQgZXh0ZW5kZWQgdGhlIGZpbGUN
Cj4gPiBmcm9tDQo+ID4gb2Zmc2V0IDQwOTYtIDgxOTEgY29tcGxldGVkIGJlZm9yZSB0aGUgd3Jp
dGUgdGhhdCB3YXMgc3VwcG9zZWQgdG8NCj4gPiBleHRlbmQgaXQgZnJvbSBvZmZzZXQgMC00MDk1
IHdhcyBwcm9jZXNzZWQgYnkgdGhlIHNlcnZlci4NCj4gDQo+ICBPdXIgaXNzdWUgYWxzbyBoYXBw
ZW5zIHdoZW4gdGhlIHdyaXRlcyBhcmUgZG9uZSBvbiB0aGUgZmlsZXNlcnZlciwNCj4gdGhvdWdo
LCBhbmQgdGhleSBvY2N1ciBldmVuIGlmIHlvdSBhbGxvdyBwbGVudHkgb2YgdGltZSBmb3IgdGhl
DQo+IHdyaXRlcw0KPiB0byBzZXR0bGUuIEkgY2FuIHJ1biBteSB0ZXN0IHByb2dyYW0gaW4gYSBt
b2RlIHdoZXJlIGl0IGV4cGxpY2l0bHkNCj4gd2FpdHMNCj4gZm9yIG1lIHRvIHRlbGwgaXQgdG8g
Y29udGludWUsIGRvIHRoZSBhcHBlbmRpbmcgdG8gdGhlIGZpbGUgb24gdGhlDQo+IGZpbGVzZXJ2
ZXIsICdzeW5jJyBvbiB0aGUgZmlsZXNlcnZlciwgd2FpdCBmaXZlIG1pbnV0ZXMsIGFuZCB0aGUg
TkZTDQo+IGNsaWVudCB3aWxsIHN0aWxsIHNlZSB0aG9zZSB6ZXJvIGJ5dGVzIHdoZW4gaXQgdHJp
ZXMgdG8gcmVhZCB0aGUgbmV3DQo+IGRhdGEuDQo+IA0KPiAoVG8gbWFrZSBzdXJlIHRoZSAnZml2
ZSBtaW51dGVzJyBiaXQgd2Fzbid0IGh5cGVyYm9sZSwgSSBqdXN0IHRlc3RlZA0KPiBpdC4pDQo+
IA0KDQpUaGF0J3MgaGFwcGVuaW5nIGJlY2F1c2Ugd2UncmUgbm90IG9wdGltaXNpbmcgZm9yIHRo
ZSBicm9rZW4gY2FzZSwgYW5kDQppbnN0ZWFkIHdlIGFzc3VtZSB0aGF0IHdlIGNhbiBjYWNoZSBk
YXRhIGZvciBhcyBsb25nIGFzIHRoZSBmaWxlIGlzDQpvcGVuIGFuZCB1bmxvY2tlZCBhcyBpbmRl
ZWQgdGhlIGNsb3NlLXRvLW9wZW4gY2FjaGUgY29uc2lzdGVuY3kgbW9kZWwNCmhhcyBhbHdheXMg
c3RhdGVkIHRoYXQgd2UgY2FuIGRvLg0KDQpUcnlpbmcgdG8gaW52YWxpZGF0ZSB0aGUgY2FjaGUg
b24gZXZlcnkgdW5leHBlY3RlZCBhdHRyaWJ1dGUgY2hhbmdlDQpsZWFkcyB0byBhIGxvdCBvZiB1
bm5lY2Vzc2FyeSBtaXNzZWQgY2FjaGVkIGhpdHMgd2hlbiBkb2luZyBvcmRpbmFyeQ0KSS9PIGJl
Y2F1c2UgaXQgbWVhbnMgd2UgdG9zcyBvdXQgdGhlIGNhY2hlIG9uIGV2ZXJ5IHJlb3JkZXJlZCB3
cml0ZSwNCmV2ZXJ5IGxpbmssIHVubGluaywgcmVuYW1lLCBldGMgb2YgdGhlIGZpbGUuDQpJT1cg
aXQgaGFzIGEgbGFyZ2UgcGVyZm9ybWFuY2UgaW1wYWN0IGZvbGxvd2luZyBhIG51bWJlciBvZiBv
cGVyYXRpb25zDQp0aGF0IGFyZSBwZXJmZWN0bHkgbGVnYWwgYW5kIGluZGVlZCBjb21tb24gdW5k
ZXIgdGhlIGNsb3NlLXRvLW9wZW4NCm1vZGVsIHdoaWxlIGZhaWxpbmcgdG8gZml4IHRoZSBwcm9i
bGVtcyBvZiB0aGUgY2xvc2UtdG8tb3BlbiB2aW9sYXRpbmcNCm1vZGVsLg0KDQotLSANClRyb25k
IE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyLCBIYW1tZXJzcGFjZQ0KdHJv
bmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNvbQ0KDQoNCg==

Re: A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

> >  Our issue also happens when the writes are done on the fileserver,
> > though, and they occur even if you allow plenty of time for the
> > writes to settle. I can run my test program in a mode where it
> > explicitly waits for me to tell it to continue, do the appending
> > to the file on the fileserver, 'sync' on the fileserver, wait five
> > minutes, and the NFS client will still see those zero bytes when it
> > tries to read the new data.
>
> That's happening because we're not optimising for the broken case, and
> instead we assume that we can cache data for as long as the file is
> open and unlocked as indeed the close-to-open cache consistency model
> has always stated that we can do.

 If I'm understanding all of this right, is what the kernel does more
or less like this: when a NFS client program closes a writeable file
(descriptor), the kernel flushes any pending writes, does a GETATTR
afterward, and declares all current cached pages fully valid 'as of'
that GETATTR result. When the file is reopened (in any mode), the kernel
GETATTRs the file again; if the GETATTR hasn't changed, the cached pages
and their contents remain valid.

 As a result, if you write to the file from another machine (including
the fileserver) before the writeable file is closed, on close the
client uses the updated GETATTR from the server but its current cached
pages. These cached pages may be out of date, but if so it is because
one violated close-to-open; you must always close any writeable file
descriptors on machine A before writing to the file on machine B (or
obtain and then release locks?).

 If a client kernel has cached pages this way, is there any simple
sequence of system calls on the client that will cause it to discard
these cached pages? Or do you need the file's GETATTR to change again,
implicitly from another machine? (I assume that changing the file's
attributes from the client with the cached pages doesn't cause it to
invalidate them, and certainly eg a 'touch' doesn't do it from the
client where it does do it from another machine.)

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDE5OjQ1IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl
Og0KPiA+ID4gIE91ciBpc3N1ZSBhbHNvIGhhcHBlbnMgd2hlbiB0aGUgd3JpdGVzIGFyZSBkb25l
IG9uIHRoZQ0KPiA+ID4gZmlsZXNlcnZlciwNCj4gPiA+IHRob3VnaCwgYW5kIHRoZXkgb2NjdXIg
ZXZlbiBpZiB5b3UgYWxsb3cgcGxlbnR5IG9mIHRpbWUgZm9yIHRoZQ0KPiA+ID4gd3JpdGVzIHRv
IHNldHRsZS4gSSBjYW4gcnVuIG15IHRlc3QgcHJvZ3JhbSBpbiBhIG1vZGUgd2hlcmUgaXQNCj4g
PiA+IGV4cGxpY2l0bHkgd2FpdHMgZm9yIG1lIHRvIHRlbGwgaXQgdG8gY29udGludWUsIGRvIHRo
ZSBhcHBlbmRpbmcNCj4gPiA+IHRvIHRoZSBmaWxlIG9uIHRoZSBmaWxlc2VydmVyLCAnc3luYycg
b24gdGhlIGZpbGVzZXJ2ZXIsIHdhaXQNCj4gPiA+IGZpdmUNCj4gPiA+IG1pbnV0ZXMsIGFuZCB0
aGUgTkZTIGNsaWVudCB3aWxsIHN0aWxsIHNlZSB0aG9zZSB6ZXJvIGJ5dGVzIHdoZW4NCj4gPiA+
IGl0DQo+ID4gPiB0cmllcyB0byByZWFkIHRoZSBuZXcgZGF0YS4NCj4gPiANCj4gPiBUaGF0J3Mg
aGFwcGVuaW5nIGJlY2F1c2Ugd2UncmUgbm90IG9wdGltaXNpbmcgZm9yIHRoZSBicm9rZW4gY2Fz
ZSwNCj4gPiBhbmQNCj4gPiBpbnN0ZWFkIHdlIGFzc3VtZSB0aGF0IHdlIGNhbiBjYWNoZSBkYXRh
IGZvciBhcyBsb25nIGFzIHRoZSBmaWxlIGlzDQo+ID4gb3BlbiBhbmQgdW5sb2NrZWQgYXMgaW5k
ZWVkIHRoZSBjbG9zZS10by1vcGVuIGNhY2hlIGNvbnNpc3RlbmN5DQo+ID4gbW9kZWwNCj4gPiBo
YXMgYWx3YXlzIHN0YXRlZCB0aGF0IHdlIGNhbiBkby4NCj4gDQo+ICBJZiBJJ20gdW5kZXJzdGFu
ZGluZyBhbGwgb2YgdGhpcyByaWdodCwgaXMgd2hhdCB0aGUga2VybmVsIGRvZXMgbW9yZQ0KPiBv
ciBsZXNzIGxpa2UgdGhpczogd2hlbiBhIE5GUyBjbGllbnQgcHJvZ3JhbSBjbG9zZXMgYSB3cml0
ZWFibGUgZmlsZQ0KPiAoZGVzY3JpcHRvciksIHRoZSBrZXJuZWwgZmx1c2hlcyBhbnkgcGVuZGlu
ZyB3cml0ZXMsIGRvZXMgYSBHRVRBVFRSDQo+IGFmdGVyd2FyZCwgYW5kIGRlY2xhcmVzIGFsbCBj
dXJyZW50IGNhY2hlZCBwYWdlcyBmdWxseSB2YWxpZCAnYXMgb2YnDQo+IHRoYXQgR0VUQVRUUiBy
ZXN1bHQuIFdoZW4gdGhlIGZpbGUgaXMgcmVvcGVuZWQgKGluIGFueSBtb2RlKSwgdGhlDQo+IGtl
cm5lbA0KPiBHRVRBVFRScyB0aGUgZmlsZSBhZ2FpbjsgaWYgdGhlIEdFVEFUVFIgaGFzbid0IGNo
YW5nZWQsIHRoZSBjYWNoZWQNCj4gcGFnZXMNCj4gYW5kIHRoZWlyIGNvbnRlbnRzIHJlbWFpbiB2
YWxpZC4NCj4gDQo+ICBBcyBhIHJlc3VsdCwgaWYgeW91IHdyaXRlIHRvIHRoZSBmaWxlIGZyb20g
YW5vdGhlciBtYWNoaW5lDQo+IChpbmNsdWRpbmcNCj4gdGhlIGZpbGVzZXJ2ZXIpIGJlZm9yZSB0
aGUgd3JpdGVhYmxlIGZpbGUgaXMgY2xvc2VkLCBvbiBjbG9zZSB0aGUNCj4gY2xpZW50IHVzZXMg
dGhlIHVwZGF0ZWQgR0VUQVRUUiBmcm9tIHRoZSBzZXJ2ZXIgYnV0IGl0cyBjdXJyZW50DQo+IGNh
Y2hlZA0KPiBwYWdlcy4gVGhlc2UgY2FjaGVkIHBhZ2VzIG1heSBiZSBvdXQgb2YgZGF0ZSwgYnV0
IGlmIHNvIGl0IGlzIGJlY2F1c2UNCj4gb25lIHZpb2xhdGVkIGNsb3NlLXRvLW9wZW47IHlvdSBt
dXN0IGFsd2F5cyBjbG9zZSBhbnkgd3JpdGVhYmxlIGZpbGUNCj4gZGVzY3JpcHRvcnMgb24gbWFj
aGluZSBBIGJlZm9yZSB3cml0aW5nIHRvIHRoZSBmaWxlIG9uIG1hY2hpbmUgQiAob3INCj4gb2J0
YWluIGFuZCB0aGVuIHJlbGVhc2UgbG9ja3M/KS4NCj4gDQo+ICBJZiBhIGNsaWVudCBrZXJuZWwg
aGFzIGNhY2hlZCBwYWdlcyB0aGlzIHdheSwgaXMgdGhlcmUgYW55IHNpbXBsZQ0KPiBzZXF1ZW5j
ZSBvZiBzeXN0ZW0gY2FsbHMgb24gdGhlIGNsaWVudCB0aGF0IHdpbGwgY2F1c2UgaXQgdG8gZGlz
Y2FyZA0KPiB0aGVzZSBjYWNoZWQgcGFnZXM/IE9yIGRvIHlvdSBuZWVkIHRoZSBmaWxlJ3MgR0VU
QVRUUiB0byBjaGFuZ2UNCj4gYWdhaW4sDQo+IGltcGxpY2l0bHkgZnJvbSBhbm90aGVyIG1hY2hp
bmU/IChJIGFzc3VtZSB0aGF0IGNoYW5naW5nIHRoZSBmaWxlJ3MNCj4gYXR0cmlidXRlcyBmcm9t
IHRoZSBjbGllbnQgd2l0aCB0aGUgY2FjaGVkIHBhZ2VzIGRvZXNuJ3QgY2F1c2UgaXQgdG8NCj4g
aW52YWxpZGF0ZSB0aGVtLCBhbmQgY2VydGFpbmx5IGVnIGEgJ3RvdWNoJyBkb2Vzbid0IGRvIGl0
IGZyb20gdGhlDQo+IGNsaWVudCB3aGVyZSBpdCBkb2VzIGRvIGl0IGZyb20gYW5vdGhlciBtYWNo
aW5lLikNCg0KVGhlcmUgYXJlIDIgd2F5cyB0byBtYW5pcHVsYXRlIHRoZSBwYWdlIGNhY2hlIGRp
cmVjdGx5IG9uIHRoZSBjbGllbnQ6DQogICAxLiBZb3UgY2FuIGNsZWFyIG91dCB0aGUgZW50aXJl
IHBhZ2UgY2FjaGUgYXMgdGhlICdyb290JyB1c2VyLCB3aXRoIHRoZQ0KICAgICAgL3Byb2Mvc3lz
L3ZtL2Ryb3BfY2FjaGVzIGludGVyZmFjZSAoc2VlICdtYW4gNSBwcm9jJykuDQogICAyLiBBbHRl
cm5hdGl2ZWx5LCB5b3UgY2FuIHVzZSBwb3NpeF9mYWR2aXNlKCkgd2l0aCB0aGUNCiAgICAgIFBP
U0lYX0ZBRFZfRE9OVE5FRUQgZmxhZyB0byBjbGVhciBvdXQgb25seSB0aGUgcGFnZXMgdGhhdCB5
b3UgdGhpbmsNCiAgICAgIGFyZSBiYWQuIE1ha2Ugc3VyZSB0byBmaXJzdCBmc3luYygpIHNvIHRo
YXQgdGhlIHBhZ2VzIGRvbid0IGdldA0KICAgICAgcGlubmVkIGluIG1lbW9yeSBieSB2aXJ0dWUg
b2YgYmVpbmcgZGlydHkgKHNlZSAnbWFuIDIgZmFkdmlzZTY0JykuDQoNCllvdSBhbHNvIGhhdmUg
dGhlIG9wdGlvbiBvZiB1c2luZyBORlMgaXRzZWxmIHRvIGltcGxpY2l0bHkgY2hhbmdlIHRoZQ0K
Y2FjaGU6DQogICAxLiBBcyB5b3Ugc2FpZCBhYm92ZSwgeW91IGNhbiBhbHNvIGNoYW5nZSB0aGUg
ZmlsZSBvbiB0aGUgc2VydmVyIHdoaWxlDQogICAgICBpdCBpcyBjbG9zZWQgb24geW91ciBjbGll
bnQsIGFuZCB0aGVuIHJlb3BlbiBpdC4NCiAgIDIuIFlvdSBjYW4gcGVyZm9ybSBhbiBPX0RJUkVD
VCB3cml0ZSBmcm9tIHRoZSBjbGllbnQgaXRzZWxmLiBCb3RoIHRob3NlDQogICAgICBvcGVyYXRp
b25zIHdpbGwgYWxzbyBpbXBseSBhIGNhY2hlIGludmFsaWRhdGlvbi4NCg0KLS0gDQpUcm9uZCBN
eWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lciwgSGFtbWVyc3BhY2UNCnRyb25k
Lm15a2xlYnVzdEBoYW1tZXJzcGFjZS5jb20NCg0KDQo=

Re: A NFS client partial file corruption problem in recent/current kernels

[Chris Siebenmann]

> >  If a client kernel has cached pages this way, is there any simple
> > sequence of system calls on the client that will cause it to discard
> > these cached pages? Or do you need the file's GETATTR to change again,
> > implicitly from another machine? (I assume that changing the file's
> > attributes from the client with the cached pages doesn't cause it to
> > invalidate them, and certainly eg a 'touch' doesn't do it from the
> > client where it does do it from another machine.)
> 
> There are 2 ways to manipulate the page cache directly on the client:
>    1. You can clear out the entire page cache as the 'root' user, with the
>       /proc/sys/vm/drop_caches interface (see 'man 5 proc').
>    2. Alternatively, you can use posix_fadvise() with the
>       POSIX_FADV_DONTNEED flag to clear out only the pages that you think
>       are bad. Make sure to first fsync() so that the pages don't get
>       pinned in memory by virtue of being dirty (see 'man 2 fadvise64').

 I just did some experiments, and on the Ubuntu 18.04 LTS version of
4.15.0, it appears that flock()'ing the file before re-reading it will
cause the kernel to not manifest the problem. I don't seem to have to
flock() the file initially when I read it before the change, and it's
sufficient to use LOCK_SH instead of LOCK_EX. (And I do have to flock()
after the change, otherwise I still see the problem even if I flock()
before.)

 Is this a supported/guaranteed behavior, or is it just lucky coincidence
that things currently work this way, much like it was happenstance
instead of design that things worked back in the 4.4.x era?

 It would be very convenient for us if flock() works around this,
because it turns out that the only reason Alpine is not flock()'ing
files is that it has an ancient 'do not use flock on Linux NFS' piece of
code deep inside it that was apparently there to work around a bug that
seems to have been fixed a decade or so ago:

   http://repo.or.cz/alpine.git/blob/HEAD:/imap/src/osdep/unix/flocklnx.c

	- cks

Re: A NFS client partial file corruption problem in recent/current kernels

[Trond Myklebust]

T24gVHVlLCAyMDE4LTA5LTExIGF0IDIzOjAzIC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl
Og0KPiA+ID4gIElmIGEgY2xpZW50IGtlcm5lbCBoYXMgY2FjaGVkIHBhZ2VzIHRoaXMgd2F5LCBp
cyB0aGVyZSBhbnkNCj4gPiA+IHNpbXBsZQ0KPiA+ID4gc2VxdWVuY2Ugb2Ygc3lzdGVtIGNhbGxz
IG9uIHRoZSBjbGllbnQgdGhhdCB3aWxsIGNhdXNlIGl0IHRvDQo+ID4gPiBkaXNjYXJkDQo+ID4g
PiB0aGVzZSBjYWNoZWQgcGFnZXM/IE9yIGRvIHlvdSBuZWVkIHRoZSBmaWxlJ3MgR0VUQVRUUiB0
byBjaGFuZ2UNCj4gPiA+IGFnYWluLA0KPiA+ID4gaW1wbGljaXRseSBmcm9tIGFub3RoZXIgbWFj
aGluZT8gKEkgYXNzdW1lIHRoYXQgY2hhbmdpbmcgdGhlDQo+ID4gPiBmaWxlJ3MNCj4gPiA+IGF0
dHJpYnV0ZXMgZnJvbSB0aGUgY2xpZW50IHdpdGggdGhlIGNhY2hlZCBwYWdlcyBkb2Vzbid0IGNh
dXNlIGl0DQo+ID4gPiB0bw0KPiA+ID4gaW52YWxpZGF0ZSB0aGVtLCBhbmQgY2VydGFpbmx5IGVn
IGEgJ3RvdWNoJyBkb2Vzbid0IGRvIGl0IGZyb20NCj4gPiA+IHRoZQ0KPiA+ID4gY2xpZW50IHdo
ZXJlIGl0IGRvZXMgZG8gaXQgZnJvbSBhbm90aGVyIG1hY2hpbmUuKQ0KPiA+IA0KPiA+IFRoZXJl
IGFyZSAyIHdheXMgdG8gbWFuaXB1bGF0ZSB0aGUgcGFnZSBjYWNoZSBkaXJlY3RseSBvbiB0aGUN
Cj4gPiBjbGllbnQ6DQo+ID4gICAgMS4gWW91IGNhbiBjbGVhciBvdXQgdGhlIGVudGlyZSBwYWdl
IGNhY2hlIGFzIHRoZSAncm9vdCcgdXNlciwNCj4gPiB3aXRoIHRoZQ0KPiA+ICAgICAgIC9wcm9j
L3N5cy92bS9kcm9wX2NhY2hlcyBpbnRlcmZhY2UgKHNlZSAnbWFuIDUgcHJvYycpLg0KPiA+ICAg
IDIuIEFsdGVybmF0aXZlbHksIHlvdSBjYW4gdXNlIHBvc2l4X2ZhZHZpc2UoKSB3aXRoIHRoZQ0K
PiA+ICAgICAgIFBPU0lYX0ZBRFZfRE9OVE5FRUQgZmxhZyB0byBjbGVhciBvdXQgb25seSB0aGUg
cGFnZXMgdGhhdCB5b3UNCj4gPiB0aGluaw0KPiA+ICAgICAgIGFyZSBiYWQuIE1ha2Ugc3VyZSB0
byBmaXJzdCBmc3luYygpIHNvIHRoYXQgdGhlIHBhZ2VzIGRvbid0DQo+ID4gZ2V0DQo+ID4gICAg
ICAgcGlubmVkIGluIG1lbW9yeSBieSB2aXJ0dWUgb2YgYmVpbmcgZGlydHkgKHNlZSAnbWFuIDIN
Cj4gPiBmYWR2aXNlNjQnKS4NCj4gDQo+ICBJIGp1c3QgZGlkIHNvbWUgZXhwZXJpbWVudHMsIGFu
ZCBvbiB0aGUgVWJ1bnR1IDE4LjA0IExUUyB2ZXJzaW9uIG9mDQo+IDQuMTUuMCwgaXQgYXBwZWFy
cyB0aGF0IGZsb2NrKCknaW5nIHRoZSBmaWxlIGJlZm9yZSByZS1yZWFkaW5nIGl0DQo+IHdpbGwN
Cj4gY2F1c2UgdGhlIGtlcm5lbCB0byBub3QgbWFuaWZlc3QgdGhlIHByb2JsZW0uIEkgZG9uJ3Qg
c2VlbSB0byBoYXZlIHRvDQo+IGZsb2NrKCkgdGhlIGZpbGUgaW5pdGlhbGx5IHdoZW4gSSByZWFk
IGl0IGJlZm9yZSB0aGUgY2hhbmdlLCBhbmQgaXQncw0KPiBzdWZmaWNpZW50IHRvIHVzZSBMT0NL
X1NIIGluc3RlYWQgb2YgTE9DS19FWC4gKEFuZCBJIGRvIGhhdmUgdG8NCj4gZmxvY2soKQ0KPiBh
ZnRlciB0aGUgY2hhbmdlLCBvdGhlcndpc2UgSSBzdGlsbCBzZWUgdGhlIHByb2JsZW0gZXZlbiBp
ZiBJIGZsb2NrKCkNCj4gYmVmb3JlLikNCj4gDQo+ICBJcyB0aGlzIGEgc3VwcG9ydGVkL2d1YXJh
bnRlZWQgYmVoYXZpb3IsIG9yIGlzIGl0IGp1c3QgbHVja3kNCj4gY29pbmNpZGVuY2UNCj4gdGhh
dCB0aGluZ3MgY3VycmVudGx5IHdvcmsgdGhpcyB3YXksIG11Y2ggbGlrZSBpdCB3YXMgaGFwcGVu
c3RhbmNlDQo+IGluc3RlYWQgb2YgZGVzaWduIHRoYXQgdGhpbmdzIHdvcmtlZCBiYWNrIGluIHRo
ZSA0LjQueCBlcmE/DQo+IA0KPiAgSXQgd291bGQgYmUgdmVyeSBjb252ZW5pZW50IGZvciB1cyBp
ZiBmbG9jaygpIHdvcmtzIGFyb3VuZCB0aGlzLA0KPiBiZWNhdXNlIGl0IHR1cm5zIG91dCB0aGF0
IHRoZSBvbmx5IHJlYXNvbiBBbHBpbmUgaXMgbm90IGZsb2NrKCknaW5nDQo+IGZpbGVzIGlzIHRo
YXQgaXQgaGFzIGFuIGFuY2llbnQgJ2RvIG5vdCB1c2UgZmxvY2sgb24gTGludXggTkZTJyBwaWVj
ZQ0KPiBvZg0KPiBjb2RlIGRlZXAgaW5zaWRlIGl0IHRoYXQgd2FzIGFwcGFyZW50bHkgdGhlcmUg
dG8gd29yayBhcm91bmQgYSBidWcNCj4gdGhhdA0KPiBzZWVtcyB0byBoYXZlIGJlZW4gZml4ZWQg
YSBkZWNhZGUgb3Igc28gYWdvOg0KPiANCj4gICAgDQo+IGh0dHA6Ly9yZXBvLm9yLmN6L2FscGlu
ZS5naXQvYmxvYi9IRUFEOi9pbWFwL3NyYy9vc2RlcC91bml4L2Zsb2NrbG54LmMNCj4gDQoNCk5v
dGhpbmcgc2hvdWxkIGJlIHN0b3BwaW5nIHlvdSBmcm9tIHVzaW5nIGVpdGhlciBmbG9jaygpIG9y
IFBPU0lYIGxvY2tzDQpvdmVyIE5GUy4gVGhlIG9uZSBkaWZmZXJlbmNlIHRoYXQgeW91IHdpbGwg
YmV0d2VlbiBORlMgYW5kIGxvY2FsDQpmaWxlc3lzdGVtcyBpcyB0aGF0IGZsb2NrKCkgbG9ja3Mg
Y2FuIGNvbmZsaWN0IHdpdGggUE9TSVggbG9ja3Mgd2hlbg0KcGVyZm9ybWVkIG92ZXIgTkZTLg0K
DQpJT1c6IElmIHlvdSBlbmFibGUgZmxvY2soKSBsb2NrcywgdGhlbiBlbnN1cmUgdGhhdCB5b3Ug
ZG8gbm90IGVuYWJsZQ0KUE9TSVggbG9ja3MgKGEuay5hLiBmY250bCgpIGxvY2tzLCBhLmsuYS4g
bG9ja2YoKSBsb2NrcykuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50
IG1haW50YWluZXIsIEhhbW1lcnNwYWNlDQp0cm9uZC5teWtsZWJ1c3RAaGFtbWVyc3BhY2UuY29t
DQoNCg0K