* Problem re-establishing GSS contexts after a server reboot @ 2016-07-19 14:51 Chuck Lever 2016-07-20 9:14 ` Adamson, Andy 0 siblings, 1 reply; 25+ messages in thread From: Chuck Lever @ 2016-07-19 14:51 UTC (permalink / raw) To: Adamson, Andy; +Cc: Linux NFS Mailing List Hi Andy- Thanks for taking the time to discuss this with me. I've copied linux-nfs to make this e-mail also an upstream bug report. As we saw in the network capture, recovery of GSS contexts after a server reboot fails in certain cases with NFSv4.0 and NFSv4.1 mount points. The reproducer is a simple program that generates one NFS WRITE periodically, run while the server repeatedly reboots (or one cluster head fails over to the other and back). The goal of the reproducer is to identify problems with state recovery without a lot of other I/O going on to clutter up the network capture. In the failing case, sec=krb5 is specified on the mount point, and the reproducer is run as root. We've found this combination fails with both NFSv4.0 and NFSv4.1. At mount time, the client establishes a GSS context for lease management operations, which is bound to the client's NFS service principal and uses GSS service "integrity." Call this GSS context 1. When the reproducer starts, a second GSS context is established for NFS operations associated with that user. Since the reproducer is running as root, this context is also bound to the client's NFS service principal, but it uses the GSS service "none" (reflecting the explicit request for "sec=krb5"). Call this GSS context 2. After the server reboots, the client re-establishes a TCP connection with the server, and performs a RENEW operation using context 1. Thanks to the server reboot, contexts 1 and 2 are now stale. The server thus rejects the RPC with RPCSEC_GSS_CTXPROBLEM. The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 NULL operation. Call this GSS context 3. Interestingly, the client does not resend the RENEW operation at this point (if it did, we wouldn't see this problem at all). The client then attempts to resume the reproducer workload. It sends an NFSv4 WRITE operation, using the first available GSS context in UID 0's credential cache, which is context 3, already bound to the client's NFS service principal. But GSS service "none" is used for this operation, since it is on behalf of the mount where sec=krb5 was specified. The RPC is accepted, but the server reports NFS4ERR_STALE_STATEID, since it has recently rebooted. The client responds by attempting state recovery. The first operation it tries is another RENEW. Since this is a lease management operation, the client looks in UID 0's credential cache again and finds the recently established context 3. It tries the RENEW operation using GSS context 3 with GSS service "integrity." The server rejects the RENEW RPC with AUTH_FAILED, and the client reports that "check lease failed" and terminates state recovery. The client re-drives the WRITE operation with the stale stateid with predictable results. The client again tries to recover state by sending a RENEW, and still uses the same GSS context 3 with service "integrity" and gets the same result. A (perhaps slow-motion) STALE_STATEID loop ensues, and the client mount point is deadlocked. Your analysis was that because the reproducer is run as root, both the reproducer's I/O operations, and lease management operations, attempt to use the same GSS context in UID 0's credential cache, but each uses different GSS services. The key issue seems to be why, when the mount is first established, the client is correctly able to establish two separate GSS contexts for UID 0; but after a server reboot, the client attempts to use the same GSS context with two different GSS services. One solution is to introduce a quick check before a context is used to see if the GSS service bound to it matches the GSS service that the caller intends to use. I'm not sure how that can be done without exposing a window where another caller requests the use of a GSS context and grabs the fresh one, before it can be used by our first caller and bound to its desired GSS service. Other solutions might be to somehow isolate the credential cache used for lease management operations, or to split credential caches by GSS service. -- Chuck Lever ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-19 14:51 Problem re-establishing GSS contexts after a server reboot Chuck Lever @ 2016-07-20 9:14 ` Adamson, Andy 2016-07-20 16:56 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Adamson, Andy @ 2016-07-20 9:14 UTC (permalink / raw) To: Chuck Lever; +Cc: Adamson, Andy, Linux NFS Mailing List DQo+IE9uIEp1bCAxOSwgMjAxNiwgYXQgMTA6NTEgQU0sIENodWNrIExldmVyIDxjaHVjay5sZXZl ckBvcmFjbGUuY29tPiB3cm90ZToNCj4gDQo+IEhpIEFuZHktDQo+IA0KPiBUaGFua3MgZm9yIHRh a2luZyB0aGUgdGltZSB0byBkaXNjdXNzIHRoaXMgd2l0aCBtZS4gSSd2ZQ0KPiBjb3BpZWQgbGlu dXgtbmZzIHRvIG1ha2UgdGhpcyBlLW1haWwgYWxzbyBhbiB1cHN0cmVhbSBidWcNCj4gcmVwb3J0 Lg0KPiANCj4gQXMgd2Ugc2F3IGluIHRoZSBuZXR3b3JrIGNhcHR1cmUsIHJlY292ZXJ5IG9mIEdT UyBjb250ZXh0cw0KPiBhZnRlciBhIHNlcnZlciByZWJvb3QgZmFpbHMgaW4gY2VydGFpbiBjYXNl cyB3aXRoIE5GU3Y0LjANCj4gYW5kIE5GU3Y0LjEgbW91bnQgcG9pbnRzLg0KPiANCj4gVGhlIHJl cHJvZHVjZXIgaXMgYSBzaW1wbGUgcHJvZ3JhbSB0aGF0IGdlbmVyYXRlcyBvbmUgTkZTDQo+IFdS SVRFIHBlcmlvZGljYWxseSwgcnVuIHdoaWxlIHRoZSBzZXJ2ZXIgcmVwZWF0ZWRseSByZWJvb3Rz DQo+IChvciBvbmUgY2x1c3RlciBoZWFkIGZhaWxzIG92ZXIgdG8gdGhlIG90aGVyIGFuZCBiYWNr KS4gVGhlDQo+IGdvYWwgb2YgdGhlIHJlcHJvZHVjZXIgaXMgdG8gaWRlbnRpZnkgcHJvYmxlbXMg d2l0aCBzdGF0ZQ0KPiByZWNvdmVyeSB3aXRob3V0IGEgbG90IG9mIG90aGVyIEkvTyBnb2luZyBv biB0byBjbHV0dGVyIHVwDQo+IHRoZSBuZXR3b3JrIGNhcHR1cmUuDQo+IA0KPiBJbiB0aGUgZmFp bGluZyBjYXNlLCBzZWM9a3JiNSBpcyBzcGVjaWZpZWQgb24gdGhlIG1vdW50DQo+IHBvaW50LCBh bmQgdGhlIHJlcHJvZHVjZXIgaXMgcnVuIGFzIHJvb3QuIFdlJ3ZlIGZvdW5kIHRoaXMNCj4gY29t YmluYXRpb24gZmFpbHMgd2l0aCBib3RoIE5GU3Y0LjAgYW5kIE5GU3Y0LjEuDQo+IA0KPiBBdCBt b3VudCB0aW1lLCB0aGUgY2xpZW50IGVzdGFibGlzaGVzIGEgR1NTIGNvbnRleHQgZm9yDQo+IGxl YXNlIG1hbmFnZW1lbnQgb3BlcmF0aW9ucywgd2hpY2ggaXMgYm91bmQgdG8gdGhlIGNsaWVudCdz DQo+IE5GUyBzZXJ2aWNlIHByaW5jaXBhbCBhbmQgdXNlcyBHU1Mgc2VydmljZSAiaW50ZWdyaXR5 LiINCj4gQ2FsbCB0aGlzIEdTUyBjb250ZXh0IDEuDQo+IA0KPiBXaGVuIHRoZSByZXByb2R1Y2Vy IHN0YXJ0cywgYSBzZWNvbmQgR1NTIGNvbnRleHQgaXMNCj4gZXN0YWJsaXNoZWQgZm9yIE5GUyBv cGVyYXRpb25zIGFzc29jaWF0ZWQgd2l0aCB0aGF0IHVzZXIuDQo+IFNpbmNlIHRoZSByZXByb2R1 Y2VyIGlzIHJ1bm5pbmcgYXMgcm9vdCwgdGhpcyBjb250ZXh0IGlzDQo+IGFsc28gYm91bmQgdG8g dGhlIGNsaWVudCdzIE5GUyBzZXJ2aWNlIHByaW5jaXBhbCwgYnV0IGl0DQo+IHVzZXMgdGhlIEdT UyBzZXJ2aWNlICJub25lIiAocmVmbGVjdGluZyB0aGUgZXhwbGljaXQNCj4gcmVxdWVzdCBmb3Ig InNlYz1rcmI1IikuIENhbGwgdGhpcyBHU1MgY29udGV4dCAyLg0KPiANCj4gQWZ0ZXIgdGhlIHNl cnZlciByZWJvb3RzLCB0aGUgY2xpZW50IHJlLWVzdGFibGlzaGVzIGEgVENQDQo+IGNvbm5lY3Rp b24gd2l0aCB0aGUgc2VydmVyLCBhbmQgcGVyZm9ybXMgYSBSRU5FVw0KPiBvcGVyYXRpb24gdXNp bmcgY29udGV4dCAxLiBUaGFua3MgdG8gdGhlIHNlcnZlciByZWJvb3QsDQo+IGNvbnRleHRzIDEg YW5kIDIgYXJlIG5vdyBzdGFsZS4gVGhlIHNlcnZlciB0aHVzIHJlamVjdHMNCj4gdGhlIFJQQyB3 aXRoIFJQQ1NFQ19HU1NfQ1RYUFJPQkxFTS4NCj4gDQo+IFRoZSBjbGllbnQgcGVyZm9ybXMgYSBH U1NfSU5JVF9TRUNfQ09OVEVYVCB2aWEgYW4gTkZTdjQNCj4gTlVMTCBvcGVyYXRpb24uIENhbGwg dGhpcyBHU1MgY29udGV4dCAzLg0KPiANCj4gSW50ZXJlc3RpbmdseSwgdGhlIGNsaWVudCBkb2Vz IG5vdCByZXNlbmQgdGhlIFJFTkVXDQo+IG9wZXJhdGlvbiBhdCB0aGlzIHBvaW50IChpZiBpdCBk aWQsIHdlIHdvdWxkbid0IHNlZSB0aGlzDQo+IHByb2JsZW0gYXQgYWxsKS4NCj4gDQo+IFRoZSBj bGllbnQgdGhlbiBhdHRlbXB0cyB0byByZXN1bWUgdGhlIHJlcHJvZHVjZXIgd29ya2xvYWQuDQo+ IEl0IHNlbmRzIGFuIE5GU3Y0IFdSSVRFIG9wZXJhdGlvbiwgdXNpbmcgdGhlIGZpcnN0IGF2YWls YWJsZQ0KPiBHU1MgY29udGV4dCBpbiBVSUQgMCdzIGNyZWRlbnRpYWwgY2FjaGUsIHdoaWNoIGlz IGNvbnRleHQgMywNCj4gYWxyZWFkeSBib3VuZCB0byB0aGUgY2xpZW50J3MgTkZTIHNlcnZpY2Ug cHJpbmNpcGFsLiBCdXQgR1NTDQo+IHNlcnZpY2UgIm5vbmUiIGlzIHVzZWQgZm9yIHRoaXMgb3Bl cmF0aW9uLCBzaW5jZSBpdCBpcyBvbg0KPiBiZWhhbGYgb2YgdGhlIG1vdW50IHdoZXJlIHNlYz1r cmI1IHdhcyBzcGVjaWZpZWQuDQo+IA0KPiBUaGUgUlBDIGlzIGFjY2VwdGVkLCBidXQgdGhlIHNl cnZlciByZXBvcnRzDQo+IE5GUzRFUlJfU1RBTEVfU1RBVEVJRCwgc2luY2UgaXQgaGFzIHJlY2Vu dGx5IHJlYm9vdGVkLg0KPiANCj4gVGhlIGNsaWVudCByZXNwb25kcyBieSBhdHRlbXB0aW5nIHN0 YXRlIHJlY292ZXJ5LiBUaGUNCj4gZmlyc3Qgb3BlcmF0aW9uIGl0IHRyaWVzIGlzIGFub3RoZXIg UkVORVcuIFNpbmNlIHRoaXMgaXMNCj4gYSBsZWFzZSBtYW5hZ2VtZW50IG9wZXJhdGlvbiwgdGhl IGNsaWVudCBsb29rcyBpbiBVSUQgMCdzDQo+IGNyZWRlbnRpYWwgY2FjaGUgYWdhaW4gYW5kIGZp bmRzIHRoZSByZWNlbnRseSBlc3RhYmxpc2hlZA0KPiBjb250ZXh0IDMuIEl0IHRyaWVzIHRoZSBS RU5FVyBvcGVyYXRpb24gdXNpbmcgR1NTIGNvbnRleHQNCj4gMyB3aXRoIEdTUyBzZXJ2aWNlICJp bnRlZ3JpdHkuIg0KPiANCj4gVGhlIHNlcnZlciByZWplY3RzIHRoZSBSRU5FVyBSUEMgd2l0aCBB VVRIX0ZBSUxFRCwgYW5kDQo+IHRoZSBjbGllbnQgcmVwb3J0cyB0aGF0ICJjaGVjayBsZWFzZSBm YWlsZWQiIGFuZA0KPiB0ZXJtaW5hdGVzIHN0YXRlIHJlY292ZXJ5Lg0KPiANCj4gVGhlIGNsaWVu dCByZS1kcml2ZXMgdGhlIFdSSVRFIG9wZXJhdGlvbiB3aXRoIHRoZSBzdGFsZQ0KPiBzdGF0ZWlk IHdpdGggcHJlZGljdGFibGUgcmVzdWx0cy4gVGhlIGNsaWVudCBhZ2FpbiB0cmllcw0KPiB0byBy ZWNvdmVyIHN0YXRlIGJ5IHNlbmRpbmcgYSBSRU5FVywgYW5kIHN0aWxsIHVzZXMgdGhlDQo+IHNh bWUgR1NTIGNvbnRleHQgMyB3aXRoIHNlcnZpY2UgImludGVncml0eSIgYW5kIGdldHMgdGhlDQo+ IHNhbWUgcmVzdWx0LiBBIChwZXJoYXBzIHNsb3ctbW90aW9uKSBTVEFMRV9TVEFURUlEIGxvb3AN Cj4gZW5zdWVzLCBhbmQgdGhlIGNsaWVudCBtb3VudCBwb2ludCBpcyBkZWFkbG9ja2VkLg0KPiAN Cj4gWW91ciBhbmFseXNpcyB3YXMgdGhhdCBiZWNhdXNlIHRoZSByZXByb2R1Y2VyIGlzIHJ1biBh cw0KPiByb290LCBib3RoIHRoZSByZXByb2R1Y2VyJ3MgSS9PIG9wZXJhdGlvbnMsIGFuZCBsZWFz ZQ0KPiBtYW5hZ2VtZW50IG9wZXJhdGlvbnMsIGF0dGVtcHQgdG8gdXNlIHRoZSBzYW1lIEdTUyBj b250ZXh0DQo+IGluIFVJRCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgYnV0IGVhY2ggdXNlcyBkaWZm ZXJlbnQgR1NTDQo+IHNlcnZpY2VzLg0KDQpBcyBSRkMyMjAzIHN0YXRlcywgIkluIGEgY3JlYXRp b24gcmVxdWVzdCwgdGhlIHNlcV9udW0gYW5kIHNlcnZpY2UgZmllbGRzIGFyZSB1bmRlZmluZWQg YW5kIGJvdGggbXVzdCBiZSBpZ25vcmVkIGJ5IHRoZSBzZXJ2ZXLigJ0NClNvIGEgY29udGV4dCBj cmVhdGlvbiByZXF1ZXN0IHdoaWxlIGtpY2tlZCBvZmYgYnkgYW4gb3BlcmF0aW9uIHdpdGggYSBz ZXJ2aWNlIGF0dGFjaGVkIChlLmcuIFdSSVRFIHVzZXMgcnBjX2dzc19zdmNfbm9uZSBhbmQgUkVO RVcgdXNlcyBycGNfZ3NzX3N2Y19pbnRlZ3JpdHkpLCBjYW4gYmUgdXNlZCBieSBlaXRoZXIgc2Vy dmljZSBsZXZlbC4NCkFGQUlDUyBhIHNpbmdsZSBHU1MgY29udGV4dCBjb3VsZCBpbiB0aGVvcnkg YmUgdXNlZCBmb3IgYWxsIHNlcnZpY2UgbGV2ZWxzLCBidXQgaW4gcHJhY3RpY2UsIEdTUyBjb250 ZXh0cyBhcmUgcmVzdHJpY3RlZCB0byBhIHNlcnZpY2UgbGV2ZWwgKGJ5IGNsaWVudD8gYnkgc2Vy dmVyPyApIG9uY2UgdGhleSBhcmUgdXNlZC4NCg0KDQo+IFRoZSBrZXkgaXNzdWUgc2VlbXMgdG8g YmUgd2h5LCB3aGVuIHRoZSBtb3VudA0KPiBpcyBmaXJzdCBlc3RhYmxpc2hlZCwgdGhlIGNsaWVu dCBpcyBjb3JyZWN0bHkgYWJsZSB0bw0KPiBlc3RhYmxpc2ggdHdvIHNlcGFyYXRlIEdTUyBjb250 ZXh0cyBmb3IgVUlEIDA7IGJ1dCBhZnRlcg0KPiBhIHNlcnZlciByZWJvb3QsIHRoZSBjbGllbnQg YXR0ZW1wdHMgdG8gdXNlIHRoZSBzYW1lIEdTUw0KPiBjb250ZXh0IHdpdGggdHdvIGRpZmZlcmVu dCBHU1Mgc2VydmljZXMuDQoNCkkgc3BlY3VsYXRlIHRoYXQgaXQgaXMgYSByYWNlIGJldHdlZW4g dGhlIFdSSVRFIGFuZCB0aGUgUkVORVcgdG8gdXNlIHRoZSBzYW1lIG5ld2x5IGNyZWF0ZWQgR1NT IGNvbnRleHQgdGhhdCBoYXMgbm90IGJlZW4gdXNlZCB5ZXQsIGFuZCBzbyBoYXMgbm8gYXNzaWdu ZWQgc2VydmljZSBsZXZlbCwgYW5kIHRoZSB0d28gcmVxdWVzdHMgcmFjZSB0byBzZXQgdGhlIHNl cnZpY2UgbGV2ZWwuDQoNCuKAlD5BbmR5DQo+IA0KPiBPbmUgc29sdXRpb24gaXMgdG8gaW50cm9k dWNlIGEgcXVpY2sgY2hlY2sgYmVmb3JlIGENCj4gY29udGV4dCBpcyB1c2VkIHRvIHNlZSBpZiB0 aGUgR1NTIHNlcnZpY2UgYm91bmQgdG8gaXQNCj4gbWF0Y2hlcyB0aGUgR1NTIHNlcnZpY2UgdGhh dCB0aGUgY2FsbGVyIGludGVuZHMgdG8gdXNlLg0KPiBJJ20gbm90IHN1cmUgaG93IHRoYXQgY2Fu IGJlIGRvbmUgd2l0aG91dCBleHBvc2luZyBhIHdpbmRvdw0KPiB3aGVyZSBhbm90aGVyIGNhbGxl ciByZXF1ZXN0cyB0aGUgdXNlIG9mIGEgR1NTIGNvbnRleHQgYW5kDQo+IGdyYWJzIHRoZSBmcmVz aCBvbmUsIGJlZm9yZSBpdCBjYW4gYmUgdXNlZCBieSBvdXIgZmlyc3QNCj4gY2FsbGVyIGFuZCBi b3VuZCB0byBpdHMgZGVzaXJlZCBHU1Mgc2VydmljZS4NCj4gDQo+IE90aGVyIHNvbHV0aW9ucyBt aWdodCBiZSB0byBzb21laG93IGlzb2xhdGUgdGhlIGNyZWRlbnRpYWwNCj4gY2FjaGUgdXNlZCBm b3IgbGVhc2UgbWFuYWdlbWVudCBvcGVyYXRpb25zLCBvciB0byBzcGxpdA0KPiBjcmVkZW50aWFs IGNhY2hlcyBieSBHU1Mgc2VydmljZS4NCj4gDQo+IA0KPiAtLQ0KPiBDaHVjayBMZXZlcg0KPiAN Cj4gDQo+IA0KDQo= ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-20 9:14 ` Adamson, Andy @ 2016-07-20 16:56 ` Olga Kornievskaia 2016-07-21 6:55 ` Chuck Lever 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-20 16:56 UTC (permalink / raw) To: Adamson, Andy; +Cc: Chuck Lever, Linux NFS Mailing List On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy <William.Adamson@netapp.com> wrote: > >> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >> Hi Andy- >> >> Thanks for taking the time to discuss this with me. I've >> copied linux-nfs to make this e-mail also an upstream bug >> report. >> >> As we saw in the network capture, recovery of GSS contexts >> after a server reboot fails in certain cases with NFSv4.0 >> and NFSv4.1 mount points. >> >> The reproducer is a simple program that generates one NFS >> WRITE periodically, run while the server repeatedly reboots >> (or one cluster head fails over to the other and back). The >> goal of the reproducer is to identify problems with state >> recovery without a lot of other I/O going on to clutter up >> the network capture. >> >> In the failing case, sec=krb5 is specified on the mount >> point, and the reproducer is run as root. We've found this >> combination fails with both NFSv4.0 and NFSv4.1. >> >> At mount time, the client establishes a GSS context for >> lease management operations, which is bound to the client's >> NFS service principal and uses GSS service "integrity." >> Call this GSS context 1. >> >> When the reproducer starts, a second GSS context is >> established for NFS operations associated with that user. >> Since the reproducer is running as root, this context is >> also bound to the client's NFS service principal, but it >> uses the GSS service "none" (reflecting the explicit >> request for "sec=krb5"). Call this GSS context 2. >> >> After the server reboots, the client re-establishes a TCP >> connection with the server, and performs a RENEW >> operation using context 1. Thanks to the server reboot, >> contexts 1 and 2 are now stale. The server thus rejects >> the RPC with RPCSEC_GSS_CTXPROBLEM. >> >> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >> NULL operation. Call this GSS context 3. >> >> Interestingly, the client does not resend the RENEW >> operation at this point (if it did, we wouldn't see this >> problem at all). >> >> The client then attempts to resume the reproducer workload. >> It sends an NFSv4 WRITE operation, using the first available >> GSS context in UID 0's credential cache, which is context 3, >> already bound to the client's NFS service principal. But GSS >> service "none" is used for this operation, since it is on >> behalf of the mount where sec=krb5 was specified. >> >> The RPC is accepted, but the server reports >> NFS4ERR_STALE_STATEID, since it has recently rebooted. >> >> The client responds by attempting state recovery. The >> first operation it tries is another RENEW. Since this is >> a lease management operation, the client looks in UID 0's >> credential cache again and finds the recently established >> context 3. It tries the RENEW operation using GSS context >> 3 with GSS service "integrity." >> >> The server rejects the RENEW RPC with AUTH_FAILED, and >> the client reports that "check lease failed" and >> terminates state recovery. >> >> The client re-drives the WRITE operation with the stale >> stateid with predictable results. The client again tries >> to recover state by sending a RENEW, and still uses the >> same GSS context 3 with service "integrity" and gets the >> same result. A (perhaps slow-motion) STALE_STATEID loop >> ensues, and the client mount point is deadlocked. >> >> Your analysis was that because the reproducer is run as >> root, both the reproducer's I/O operations, and lease >> management operations, attempt to use the same GSS context >> in UID 0's credential cache, but each uses different GSS >> services. > > As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” > So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. > AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. > > >> The key issue seems to be why, when the mount >> is first established, the client is correctly able to >> establish two separate GSS contexts for UID 0; but after >> a server reboot, the client attempts to use the same GSS >> context with two different GSS services. > > I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. I agree with Andy. It must be a tight race. I have tried to reproduce your scenario and in my tests of rebooting the server all recover correctly. In my case, if RENEW was the one hitting the AUTH_ERR then the new context is established and then RENEW using integrity service is retried with the new context which gets ERR_STALE_CLIENTID which then client recovers from. If it's an operation (I have a GETATTR) that gets AUTH_ERR, then it gets new context and is retried using none service. Then RENEW gets its own AUTH_ERR as it uses a different context, a new context is gotten, RENEW is retried over integrity and gets ERR_STALE_CLIENTID which it recovers from. > > —>Andy >> >> One solution is to introduce a quick check before a >> context is used to see if the GSS service bound to it >> matches the GSS service that the caller intends to use. >> I'm not sure how that can be done without exposing a window >> where another caller requests the use of a GSS context and >> grabs the fresh one, before it can be used by our first >> caller and bound to its desired GSS service. >> >> Other solutions might be to somehow isolate the credential >> cache used for lease management operations, or to split >> credential caches by GSS service. >> >> >> -- >> Chuck Lever >> >> >> > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-20 16:56 ` Olga Kornievskaia @ 2016-07-21 6:55 ` Chuck Lever 2016-07-21 16:04 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Chuck Lever @ 2016-07-21 6:55 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: Adamson, Andy, Linux NFS Mailing List > On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > > On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy > <William.Adamson@netapp.com> wrote: >> >>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>> >>> Hi Andy- >>> >>> Thanks for taking the time to discuss this with me. I've >>> copied linux-nfs to make this e-mail also an upstream bug >>> report. >>> >>> As we saw in the network capture, recovery of GSS contexts >>> after a server reboot fails in certain cases with NFSv4.0 >>> and NFSv4.1 mount points. >>> >>> The reproducer is a simple program that generates one NFS >>> WRITE periodically, run while the server repeatedly reboots >>> (or one cluster head fails over to the other and back). The >>> goal of the reproducer is to identify problems with state >>> recovery without a lot of other I/O going on to clutter up >>> the network capture. >>> >>> In the failing case, sec=krb5 is specified on the mount >>> point, and the reproducer is run as root. We've found this >>> combination fails with both NFSv4.0 and NFSv4.1. >>> >>> At mount time, the client establishes a GSS context for >>> lease management operations, which is bound to the client's >>> NFS service principal and uses GSS service "integrity." >>> Call this GSS context 1. >>> >>> When the reproducer starts, a second GSS context is >>> established for NFS operations associated with that user. >>> Since the reproducer is running as root, this context is >>> also bound to the client's NFS service principal, but it >>> uses the GSS service "none" (reflecting the explicit >>> request for "sec=krb5"). Call this GSS context 2. >>> >>> After the server reboots, the client re-establishes a TCP >>> connection with the server, and performs a RENEW >>> operation using context 1. Thanks to the server reboot, >>> contexts 1 and 2 are now stale. The server thus rejects >>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>> >>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>> NULL operation. Call this GSS context 3. >>> >>> Interestingly, the client does not resend the RENEW >>> operation at this point (if it did, we wouldn't see this >>> problem at all). >>> >>> The client then attempts to resume the reproducer workload. >>> It sends an NFSv4 WRITE operation, using the first available >>> GSS context in UID 0's credential cache, which is context 3, >>> already bound to the client's NFS service principal. But GSS >>> service "none" is used for this operation, since it is on >>> behalf of the mount where sec=krb5 was specified. >>> >>> The RPC is accepted, but the server reports >>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>> >>> The client responds by attempting state recovery. The >>> first operation it tries is another RENEW. Since this is >>> a lease management operation, the client looks in UID 0's >>> credential cache again and finds the recently established >>> context 3. It tries the RENEW operation using GSS context >>> 3 with GSS service "integrity." >>> >>> The server rejects the RENEW RPC with AUTH_FAILED, and >>> the client reports that "check lease failed" and >>> terminates state recovery. >>> >>> The client re-drives the WRITE operation with the stale >>> stateid with predictable results. The client again tries >>> to recover state by sending a RENEW, and still uses the >>> same GSS context 3 with service "integrity" and gets the >>> same result. A (perhaps slow-motion) STALE_STATEID loop >>> ensues, and the client mount point is deadlocked. >>> >>> Your analysis was that because the reproducer is run as >>> root, both the reproducer's I/O operations, and lease >>> management operations, attempt to use the same GSS context >>> in UID 0's credential cache, but each uses different GSS >>> services. >> >> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >> >> >>> The key issue seems to be why, when the mount >>> is first established, the client is correctly able to >>> establish two separate GSS contexts for UID 0; but after >>> a server reboot, the client attempts to use the same GSS >>> context with two different GSS services. >> >> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. > > I agree with Andy. It must be a tight race. In one capture I see something like this after the server restarts: SYN SYN, ACK ACK C WRITE C SEQUENCE R WRITE -> CTX_PROBLEM R SEQUENCE -> CTX_PROBLEM C NULL (KRB5_AP_REQ) R NULL (KRB5_AP_REP) C WRITE C SEQUENCE R WRITE -> NFS4ERR_STALE_STATEID R SEQUENCE -> AUTH_FAILED Andy's theory neatly explains this behavior. > I have tried to reproduce > your scenario and in my tests of rebooting the server all recover > correctly. In my case, if RENEW was the one hitting the AUTH_ERR then > the new context is established and then RENEW using integrity service > is retried with the new context which gets ERR_STALE_CLIENTID which > then client recovers from. If it's an operation (I have a GETATTR) > that gets AUTH_ERR, then it gets new context and is retried using none > service. Then RENEW gets its own AUTH_ERR as it uses a different > context, a new context is gotten, RENEW is retried over integrity and > gets ERR_STALE_CLIENTID which it recovers from. If one operation is allowed to complete, then the other will always recognize that another fresh GSS context is needed. If two are sent at the same time, they race and one always fails. Helen's test includes a second idle mount point (sec=krb5i) and maybe that is needed to trigger the race? >> —>Andy >>> >>> One solution is to introduce a quick check before a >>> context is used to see if the GSS service bound to it >>> matches the GSS service that the caller intends to use. >>> I'm not sure how that can be done without exposing a window >>> where another caller requests the use of a GSS context and >>> grabs the fresh one, before it can be used by our first >>> caller and bound to its desired GSS service. >>> >>> Other solutions might be to somehow isolate the credential >>> cache used for lease management operations, or to split >>> credential caches by GSS service. >>> >>> >>> -- >>> Chuck Lever >>> >>> >>> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-21 6:55 ` Chuck Lever @ 2016-07-21 16:04 ` Olga Kornievskaia 2016-07-21 17:56 ` Chuck Lever 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-21 16:04 UTC (permalink / raw) To: Chuck Lever; +Cc: Adamson, Andy, Linux NFS Mailing List On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > >> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >> <William.Adamson@netapp.com> wrote: >>> >>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>> >>>> Hi Andy- >>>> >>>> Thanks for taking the time to discuss this with me. I've >>>> copied linux-nfs to make this e-mail also an upstream bug >>>> report. >>>> >>>> As we saw in the network capture, recovery of GSS contexts >>>> after a server reboot fails in certain cases with NFSv4.0 >>>> and NFSv4.1 mount points. >>>> >>>> The reproducer is a simple program that generates one NFS >>>> WRITE periodically, run while the server repeatedly reboots >>>> (or one cluster head fails over to the other and back). The >>>> goal of the reproducer is to identify problems with state >>>> recovery without a lot of other I/O going on to clutter up >>>> the network capture. >>>> >>>> In the failing case, sec=krb5 is specified on the mount >>>> point, and the reproducer is run as root. We've found this >>>> combination fails with both NFSv4.0 and NFSv4.1. >>>> >>>> At mount time, the client establishes a GSS context for >>>> lease management operations, which is bound to the client's >>>> NFS service principal and uses GSS service "integrity." >>>> Call this GSS context 1. >>>> >>>> When the reproducer starts, a second GSS context is >>>> established for NFS operations associated with that user. >>>> Since the reproducer is running as root, this context is >>>> also bound to the client's NFS service principal, but it >>>> uses the GSS service "none" (reflecting the explicit >>>> request for "sec=krb5"). Call this GSS context 2. >>>> >>>> After the server reboots, the client re-establishes a TCP >>>> connection with the server, and performs a RENEW >>>> operation using context 1. Thanks to the server reboot, >>>> contexts 1 and 2 are now stale. The server thus rejects >>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>> >>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>> NULL operation. Call this GSS context 3. >>>> >>>> Interestingly, the client does not resend the RENEW >>>> operation at this point (if it did, we wouldn't see this >>>> problem at all). >>>> >>>> The client then attempts to resume the reproducer workload. >>>> It sends an NFSv4 WRITE operation, using the first available >>>> GSS context in UID 0's credential cache, which is context 3, >>>> already bound to the client's NFS service principal. But GSS >>>> service "none" is used for this operation, since it is on >>>> behalf of the mount where sec=krb5 was specified. >>>> >>>> The RPC is accepted, but the server reports >>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>> >>>> The client responds by attempting state recovery. The >>>> first operation it tries is another RENEW. Since this is >>>> a lease management operation, the client looks in UID 0's >>>> credential cache again and finds the recently established >>>> context 3. It tries the RENEW operation using GSS context >>>> 3 with GSS service "integrity." >>>> >>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>> the client reports that "check lease failed" and >>>> terminates state recovery. >>>> >>>> The client re-drives the WRITE operation with the stale >>>> stateid with predictable results. The client again tries >>>> to recover state by sending a RENEW, and still uses the >>>> same GSS context 3 with service "integrity" and gets the >>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>> ensues, and the client mount point is deadlocked. >>>> >>>> Your analysis was that because the reproducer is run as >>>> root, both the reproducer's I/O operations, and lease >>>> management operations, attempt to use the same GSS context >>>> in UID 0's credential cache, but each uses different GSS >>>> services. >>> >>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>> >>> >>>> The key issue seems to be why, when the mount >>>> is first established, the client is correctly able to >>>> establish two separate GSS contexts for UID 0; but after >>>> a server reboot, the client attempts to use the same GSS >>>> context with two different GSS services. >>> >>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >> >> I agree with Andy. It must be a tight race. > > In one capture I see something like this after > the server restarts: > > SYN > SYN, ACK > ACK > C WRITE > C SEQUENCE > R WRITE -> CTX_PROBLEM > R SEQUENCE -> CTX_PROBLEM > C NULL (KRB5_AP_REQ) > R NULL (KRB5_AP_REP) > C WRITE > C SEQUENCE > R WRITE -> NFS4ERR_STALE_STATEID > R SEQUENCE -> AUTH_FAILED > > Andy's theory neatly explains this behavior. > > >> I have tried to reproduce >> your scenario and in my tests of rebooting the server all recover >> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >> the new context is established and then RENEW using integrity service >> is retried with the new context which gets ERR_STALE_CLIENTID which >> then client recovers from. If it's an operation (I have a GETATTR) >> that gets AUTH_ERR, then it gets new context and is retried using none >> service. Then RENEW gets its own AUTH_ERR as it uses a different >> context, a new context is gotten, RENEW is retried over integrity and >> gets ERR_STALE_CLIENTID which it recovers from. > > If one operation is allowed to complete, then > the other will always recognize that another > fresh GSS context is needed. If two are sent > at the same time, they race and one always > fails. > > Helen's test includes a second idle mount point > (sec=krb5i) and maybe that is needed to trigger > the race? > Chuck, any chance to get "rpcdebug -m rpc auth" output during the failure (gssd optionally) (i realize that it might alter the timings and not hit the issue but worth a shot)? > >>> —>Andy >>>> >>>> One solution is to introduce a quick check before a >>>> context is used to see if the GSS service bound to it >>>> matches the GSS service that the caller intends to use. >>>> I'm not sure how that can be done without exposing a window >>>> where another caller requests the use of a GSS context and >>>> grabs the fresh one, before it can be used by our first >>>> caller and bound to its desired GSS service. >>>> >>>> Other solutions might be to somehow isolate the credential >>>> cache used for lease management operations, or to split >>>> credential caches by GSS service. >>>> >>>> >>>> -- >>>> Chuck Lever >>>> >>>> >>>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > Chuck Lever > > > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-21 16:04 ` Olga Kornievskaia @ 2016-07-21 17:56 ` Chuck Lever 2016-07-21 19:54 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Chuck Lever @ 2016-07-21 17:56 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: Adamson, Andy, Linux NFS Mailing List > On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > > On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>> >>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>> <William.Adamson@netapp.com> wrote: >>>> >>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>> >>>>> Hi Andy- >>>>> >>>>> Thanks for taking the time to discuss this with me. I've >>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>> report. >>>>> >>>>> As we saw in the network capture, recovery of GSS contexts >>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>> and NFSv4.1 mount points. >>>>> >>>>> The reproducer is a simple program that generates one NFS >>>>> WRITE periodically, run while the server repeatedly reboots >>>>> (or one cluster head fails over to the other and back). The >>>>> goal of the reproducer is to identify problems with state >>>>> recovery without a lot of other I/O going on to clutter up >>>>> the network capture. >>>>> >>>>> In the failing case, sec=krb5 is specified on the mount >>>>> point, and the reproducer is run as root. We've found this >>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>> >>>>> At mount time, the client establishes a GSS context for >>>>> lease management operations, which is bound to the client's >>>>> NFS service principal and uses GSS service "integrity." >>>>> Call this GSS context 1. >>>>> >>>>> When the reproducer starts, a second GSS context is >>>>> established for NFS operations associated with that user. >>>>> Since the reproducer is running as root, this context is >>>>> also bound to the client's NFS service principal, but it >>>>> uses the GSS service "none" (reflecting the explicit >>>>> request for "sec=krb5"). Call this GSS context 2. >>>>> >>>>> After the server reboots, the client re-establishes a TCP >>>>> connection with the server, and performs a RENEW >>>>> operation using context 1. Thanks to the server reboot, >>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>> >>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>> NULL operation. Call this GSS context 3. >>>>> >>>>> Interestingly, the client does not resend the RENEW >>>>> operation at this point (if it did, we wouldn't see this >>>>> problem at all). >>>>> >>>>> The client then attempts to resume the reproducer workload. >>>>> It sends an NFSv4 WRITE operation, using the first available >>>>> GSS context in UID 0's credential cache, which is context 3, >>>>> already bound to the client's NFS service principal. But GSS >>>>> service "none" is used for this operation, since it is on >>>>> behalf of the mount where sec=krb5 was specified. >>>>> >>>>> The RPC is accepted, but the server reports >>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>> >>>>> The client responds by attempting state recovery. The >>>>> first operation it tries is another RENEW. Since this is >>>>> a lease management operation, the client looks in UID 0's >>>>> credential cache again and finds the recently established >>>>> context 3. It tries the RENEW operation using GSS context >>>>> 3 with GSS service "integrity." >>>>> >>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>> the client reports that "check lease failed" and >>>>> terminates state recovery. >>>>> >>>>> The client re-drives the WRITE operation with the stale >>>>> stateid with predictable results. The client again tries >>>>> to recover state by sending a RENEW, and still uses the >>>>> same GSS context 3 with service "integrity" and gets the >>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>> ensues, and the client mount point is deadlocked. >>>>> >>>>> Your analysis was that because the reproducer is run as >>>>> root, both the reproducer's I/O operations, and lease >>>>> management operations, attempt to use the same GSS context >>>>> in UID 0's credential cache, but each uses different GSS >>>>> services. >>>> >>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>> >>>> >>>>> The key issue seems to be why, when the mount >>>>> is first established, the client is correctly able to >>>>> establish two separate GSS contexts for UID 0; but after >>>>> a server reboot, the client attempts to use the same GSS >>>>> context with two different GSS services. >>>> >>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>> >>> I agree with Andy. It must be a tight race. >> >> In one capture I see something like this after >> the server restarts: >> >> SYN >> SYN, ACK >> ACK >> C WRITE >> C SEQUENCE >> R WRITE -> CTX_PROBLEM >> R SEQUENCE -> CTX_PROBLEM >> C NULL (KRB5_AP_REQ) >> R NULL (KRB5_AP_REP) >> C WRITE >> C SEQUENCE >> R WRITE -> NFS4ERR_STALE_STATEID >> R SEQUENCE -> AUTH_FAILED >> >> Andy's theory neatly explains this behavior. >> >> >>> I have tried to reproduce >>> your scenario and in my tests of rebooting the server all recover >>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>> the new context is established and then RENEW using integrity service >>> is retried with the new context which gets ERR_STALE_CLIENTID which >>> then client recovers from. If it's an operation (I have a GETATTR) >>> that gets AUTH_ERR, then it gets new context and is retried using none >>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>> context, a new context is gotten, RENEW is retried over integrity and >>> gets ERR_STALE_CLIENTID which it recovers from. >> >> If one operation is allowed to complete, then >> the other will always recognize that another >> fresh GSS context is needed. If two are sent >> at the same time, they race and one always >> fails. >> >> Helen's test includes a second idle mount point >> (sec=krb5i) and maybe that is needed to trigger >> the race? >> > > Chuck, any chance to get "rpcdebug -m rpc auth" output during the > failure (gssd optionally) (i realize that it might alter the timings > and not hit the issue but worth a shot)? I'm sure that's fine. An internal tester hit this, not a customer, so I will ask. I agree, though, that timing might be a problem: these systems all have real serial consoles via iLOM, so /v/l/m traffic does bring everything to a standstill. Meanwhile, what's you're opinion about AUTH_FAILED? Should the server return RPCSEC_GSS_CTXPROBLEM in this case instead? If it did, do you think the Linux client would recover by creating a replacement GSS context? >>>> —>Andy >>>>> >>>>> One solution is to introduce a quick check before a >>>>> context is used to see if the GSS service bound to it >>>>> matches the GSS service that the caller intends to use. >>>>> I'm not sure how that can be done without exposing a window >>>>> where another caller requests the use of a GSS context and >>>>> grabs the fresh one, before it can be used by our first >>>>> caller and bound to its desired GSS service. >>>>> >>>>> Other solutions might be to somehow isolate the credential >>>>> cache used for lease management operations, or to split >>>>> credential caches by GSS service. >>>>> >>>>> >>>>> -- >>>>> Chuck Lever >>>>> >>>>> >>>>> >>>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- >> Chuck Lever >> >> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-21 17:56 ` Chuck Lever @ 2016-07-21 19:54 ` Olga Kornievskaia 2016-07-21 20:46 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-21 19:54 UTC (permalink / raw) To: Chuck Lever; +Cc: Adamson, Andy, Linux NFS Mailing List On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > >> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>> >>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> >>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>> <William.Adamson@netapp.com> wrote: >>>>> >>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>> >>>>>> Hi Andy- >>>>>> >>>>>> Thanks for taking the time to discuss this with me. I've >>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>> report. >>>>>> >>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>> and NFSv4.1 mount points. >>>>>> >>>>>> The reproducer is a simple program that generates one NFS >>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>> (or one cluster head fails over to the other and back). The >>>>>> goal of the reproducer is to identify problems with state >>>>>> recovery without a lot of other I/O going on to clutter up >>>>>> the network capture. >>>>>> >>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>> point, and the reproducer is run as root. We've found this >>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>> >>>>>> At mount time, the client establishes a GSS context for >>>>>> lease management operations, which is bound to the client's >>>>>> NFS service principal and uses GSS service "integrity." >>>>>> Call this GSS context 1. >>>>>> >>>>>> When the reproducer starts, a second GSS context is >>>>>> established for NFS operations associated with that user. >>>>>> Since the reproducer is running as root, this context is >>>>>> also bound to the client's NFS service principal, but it >>>>>> uses the GSS service "none" (reflecting the explicit >>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>> >>>>>> After the server reboots, the client re-establishes a TCP >>>>>> connection with the server, and performs a RENEW >>>>>> operation using context 1. Thanks to the server reboot, >>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>> >>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>> NULL operation. Call this GSS context 3. >>>>>> >>>>>> Interestingly, the client does not resend the RENEW >>>>>> operation at this point (if it did, we wouldn't see this >>>>>> problem at all). >>>>>> >>>>>> The client then attempts to resume the reproducer workload. >>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>> already bound to the client's NFS service principal. But GSS >>>>>> service "none" is used for this operation, since it is on >>>>>> behalf of the mount where sec=krb5 was specified. >>>>>> >>>>>> The RPC is accepted, but the server reports >>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>> >>>>>> The client responds by attempting state recovery. The >>>>>> first operation it tries is another RENEW. Since this is >>>>>> a lease management operation, the client looks in UID 0's >>>>>> credential cache again and finds the recently established >>>>>> context 3. It tries the RENEW operation using GSS context >>>>>> 3 with GSS service "integrity." >>>>>> >>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>> the client reports that "check lease failed" and >>>>>> terminates state recovery. >>>>>> >>>>>> The client re-drives the WRITE operation with the stale >>>>>> stateid with predictable results. The client again tries >>>>>> to recover state by sending a RENEW, and still uses the >>>>>> same GSS context 3 with service "integrity" and gets the >>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>> ensues, and the client mount point is deadlocked. >>>>>> >>>>>> Your analysis was that because the reproducer is run as >>>>>> root, both the reproducer's I/O operations, and lease >>>>>> management operations, attempt to use the same GSS context >>>>>> in UID 0's credential cache, but each uses different GSS >>>>>> services. >>>>> >>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>> >>>>> >>>>>> The key issue seems to be why, when the mount >>>>>> is first established, the client is correctly able to >>>>>> establish two separate GSS contexts for UID 0; but after >>>>>> a server reboot, the client attempts to use the same GSS >>>>>> context with two different GSS services. >>>>> >>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>> >>>> I agree with Andy. It must be a tight race. >>> >>> In one capture I see something like this after >>> the server restarts: >>> >>> SYN >>> SYN, ACK >>> ACK >>> C WRITE >>> C SEQUENCE >>> R WRITE -> CTX_PROBLEM >>> R SEQUENCE -> CTX_PROBLEM >>> C NULL (KRB5_AP_REQ) >>> R NULL (KRB5_AP_REP) >>> C WRITE >>> C SEQUENCE >>> R WRITE -> NFS4ERR_STALE_STATEID >>> R SEQUENCE -> AUTH_FAILED >>> >>> Andy's theory neatly explains this behavior. >>> >>> >>>> I have tried to reproduce >>>> your scenario and in my tests of rebooting the server all recover >>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>> the new context is established and then RENEW using integrity service >>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>> then client recovers from. If it's an operation (I have a GETATTR) >>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>> context, a new context is gotten, RENEW is retried over integrity and >>>> gets ERR_STALE_CLIENTID which it recovers from. >>> >>> If one operation is allowed to complete, then >>> the other will always recognize that another >>> fresh GSS context is needed. If two are sent >>> at the same time, they race and one always >>> fails. >>> >>> Helen's test includes a second idle mount point >>> (sec=krb5i) and maybe that is needed to trigger >>> the race? >>> >> >> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >> failure (gssd optionally) (i realize that it might alter the timings >> and not hit the issue but worth a shot)? > > I'm sure that's fine. An internal tester hit this, > not a customer, so I will ask. > > I agree, though, that timing might be a problem: > these systems all have real serial consoles via > iLOM, so /v/l/m traffic does bring everything to > a standstill. > > Meanwhile, what's you're opinion about AUTH_FAILED? > Should the server return RPCSEC_GSS_CTXPROBLEM > in this case instead? If it did, do you think > the Linux client would recover by creating a > replacement GSS context? Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will fail with EIO error and will not try to create a new GSS context. So yes, I believe it would help if the server returns any of the following errors: case RPC_AUTH_REJECTEDCRED: case RPC_AUTH_REJECTEDVERF: case RPCSEC_GSS_CREDPROBLEM: case RPCSEC_GSS_CTXPROBLEM: then the client will recreate the context. > > > >>>>> —>Andy >>>>>> >>>>>> One solution is to introduce a quick check before a >>>>>> context is used to see if the GSS service bound to it >>>>>> matches the GSS service that the caller intends to use. >>>>>> I'm not sure how that can be done without exposing a window >>>>>> where another caller requests the use of a GSS context and >>>>>> grabs the fresh one, before it can be used by our first >>>>>> caller and bound to its desired GSS service. >>>>>> >>>>>> Other solutions might be to somehow isolate the credential >>>>>> cache used for lease management operations, or to split >>>>>> credential caches by GSS service. >>>>>> >>>>>> >>>>>> -- >>>>>> Chuck Lever >>>>>> >>>>>> >>>>>> >>>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> -- >>> Chuck Lever >>> >>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > Chuck Lever > > > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-21 19:54 ` Olga Kornievskaia @ 2016-07-21 20:46 ` Olga Kornievskaia 2016-07-21 21:32 ` Chuck Lever 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-21 20:46 UTC (permalink / raw) To: Chuck Lever; +Cc: Adamson, Andy, Linux NFS Mailing List On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>> >>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>> >>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>> >>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>> <William.Adamson@netapp.com> wrote: >>>>>> >>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>> >>>>>>> Hi Andy- >>>>>>> >>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>> report. >>>>>>> >>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>> and NFSv4.1 mount points. >>>>>>> >>>>>>> The reproducer is a simple program that generates one NFS >>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>> (or one cluster head fails over to the other and back). The >>>>>>> goal of the reproducer is to identify problems with state >>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>> the network capture. >>>>>>> >>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>> point, and the reproducer is run as root. We've found this >>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>> >>>>>>> At mount time, the client establishes a GSS context for >>>>>>> lease management operations, which is bound to the client's >>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>> Call this GSS context 1. >>>>>>> >>>>>>> When the reproducer starts, a second GSS context is >>>>>>> established for NFS operations associated with that user. >>>>>>> Since the reproducer is running as root, this context is >>>>>>> also bound to the client's NFS service principal, but it >>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>> >>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>> connection with the server, and performs a RENEW >>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>> >>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>> NULL operation. Call this GSS context 3. >>>>>>> >>>>>>> Interestingly, the client does not resend the RENEW >>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>> problem at all). >>>>>>> >>>>>>> The client then attempts to resume the reproducer workload. >>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>> service "none" is used for this operation, since it is on >>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>> >>>>>>> The RPC is accepted, but the server reports >>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>> >>>>>>> The client responds by attempting state recovery. The >>>>>>> first operation it tries is another RENEW. Since this is >>>>>>> a lease management operation, the client looks in UID 0's >>>>>>> credential cache again and finds the recently established >>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>> 3 with GSS service "integrity." >>>>>>> >>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>> the client reports that "check lease failed" and >>>>>>> terminates state recovery. >>>>>>> >>>>>>> The client re-drives the WRITE operation with the stale >>>>>>> stateid with predictable results. The client again tries >>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>> ensues, and the client mount point is deadlocked. >>>>>>> >>>>>>> Your analysis was that because the reproducer is run as >>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>> management operations, attempt to use the same GSS context >>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>> services. >>>>>> >>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>> >>>>>> >>>>>>> The key issue seems to be why, when the mount >>>>>>> is first established, the client is correctly able to >>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>> context with two different GSS services. >>>>>> >>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>> >>>>> I agree with Andy. It must be a tight race. >>>> >>>> In one capture I see something like this after >>>> the server restarts: >>>> >>>> SYN >>>> SYN, ACK >>>> ACK >>>> C WRITE >>>> C SEQUENCE >>>> R WRITE -> CTX_PROBLEM >>>> R SEQUENCE -> CTX_PROBLEM >>>> C NULL (KRB5_AP_REQ) >>>> R NULL (KRB5_AP_REP) >>>> C WRITE >>>> C SEQUENCE >>>> R WRITE -> NFS4ERR_STALE_STATEID >>>> R SEQUENCE -> AUTH_FAILED >>>> >>>> Andy's theory neatly explains this behavior. >>>> >>>> >>>>> I have tried to reproduce >>>>> your scenario and in my tests of rebooting the server all recover >>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>> the new context is established and then RENEW using integrity service >>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>> >>>> If one operation is allowed to complete, then >>>> the other will always recognize that another >>>> fresh GSS context is needed. If two are sent >>>> at the same time, they race and one always >>>> fails. >>>> >>>> Helen's test includes a second idle mount point >>>> (sec=krb5i) and maybe that is needed to trigger >>>> the race? >>>> >>> >>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>> failure (gssd optionally) (i realize that it might alter the timings >>> and not hit the issue but worth a shot)? >> >> I'm sure that's fine. An internal tester hit this, >> not a customer, so I will ask. >> >> I agree, though, that timing might be a problem: >> these systems all have real serial consoles via >> iLOM, so /v/l/m traffic does bring everything to >> a standstill. >> >> Meanwhile, what's you're opinion about AUTH_FAILED? >> Should the server return RPCSEC_GSS_CTXPROBLEM >> in this case instead? If it did, do you think >> the Linux client would recover by creating a >> replacement GSS context? > > Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client > receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will > fail with EIO error and will not try to create a new GSS context. So > yes, I believe it would help if the server returns any of the > following errors: > case RPC_AUTH_REJECTEDCRED: > case RPC_AUTH_REJECTEDVERF: > case RPCSEC_GSS_CREDPROBLEM: > case RPCSEC_GSS_CTXPROBLEM: > > then the client will recreate the context. > Also in my testing, I can see that credential cache is per gss flavor. Just to check, what kernel version is this problem encountered on (I know you said upstream) but I just want to double check so that I can look at the correct source code. Thanks. > >> >> >> >>>>>> —>Andy >>>>>>> >>>>>>> One solution is to introduce a quick check before a >>>>>>> context is used to see if the GSS service bound to it >>>>>>> matches the GSS service that the caller intends to use. >>>>>>> I'm not sure how that can be done without exposing a window >>>>>>> where another caller requests the use of a GSS context and >>>>>>> grabs the fresh one, before it can be used by our first >>>>>>> caller and bound to its desired GSS service. >>>>>>> >>>>>>> Other solutions might be to somehow isolate the credential >>>>>>> cache used for lease management operations, or to split >>>>>>> credential caches by GSS service. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Chuck Lever >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> -- >>>> Chuck Lever >>>> >>>> >>>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- >> Chuck Lever >> >> >> ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-21 20:46 ` Olga Kornievskaia @ 2016-07-21 21:32 ` Chuck Lever 2016-07-25 18:18 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Chuck Lever @ 2016-07-21 21:32 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: Adamson, Andy, Linux NFS Mailing List > On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>> >>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> >>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>> >>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>> >>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>> <William.Adamson@netapp.com> wrote: >>>>>>> >>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>> >>>>>>>> Hi Andy- >>>>>>>> >>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>> report. >>>>>>>> >>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>> and NFSv4.1 mount points. >>>>>>>> >>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>> the network capture. >>>>>>>> >>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>> >>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>> lease management operations, which is bound to the client's >>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>> Call this GSS context 1. >>>>>>>> >>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>> established for NFS operations associated with that user. >>>>>>>> Since the reproducer is running as root, this context is >>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>> >>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>> connection with the server, and performs a RENEW >>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>> >>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>> >>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>> problem at all). >>>>>>>> >>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>> service "none" is used for this operation, since it is on >>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>> >>>>>>>> The RPC is accepted, but the server reports >>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>> >>>>>>>> The client responds by attempting state recovery. The >>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>> credential cache again and finds the recently established >>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>> 3 with GSS service "integrity." >>>>>>>> >>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>> the client reports that "check lease failed" and >>>>>>>> terminates state recovery. >>>>>>>> >>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>> stateid with predictable results. The client again tries >>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>> >>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>> management operations, attempt to use the same GSS context >>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>> services. >>>>>>> >>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>> >>>>>>> >>>>>>>> The key issue seems to be why, when the mount >>>>>>>> is first established, the client is correctly able to >>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>> context with two different GSS services. >>>>>>> >>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>> >>>>>> I agree with Andy. It must be a tight race. >>>>> >>>>> In one capture I see something like this after >>>>> the server restarts: >>>>> >>>>> SYN >>>>> SYN, ACK >>>>> ACK >>>>> C WRITE >>>>> C SEQUENCE >>>>> R WRITE -> CTX_PROBLEM >>>>> R SEQUENCE -> CTX_PROBLEM >>>>> C NULL (KRB5_AP_REQ) >>>>> R NULL (KRB5_AP_REP) >>>>> C WRITE >>>>> C SEQUENCE >>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>> R SEQUENCE -> AUTH_FAILED >>>>> >>>>> Andy's theory neatly explains this behavior. >>>>> >>>>> >>>>>> I have tried to reproduce >>>>>> your scenario and in my tests of rebooting the server all recover >>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>> the new context is established and then RENEW using integrity service >>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>> >>>>> If one operation is allowed to complete, then >>>>> the other will always recognize that another >>>>> fresh GSS context is needed. If two are sent >>>>> at the same time, they race and one always >>>>> fails. >>>>> >>>>> Helen's test includes a second idle mount point >>>>> (sec=krb5i) and maybe that is needed to trigger >>>>> the race? >>>> >>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>> failure (gssd optionally) (i realize that it might alter the timings >>>> and not hit the issue but worth a shot)? >>> >>> I'm sure that's fine. An internal tester hit this, >>> not a customer, so I will ask. >>> >>> I agree, though, that timing might be a problem: >>> these systems all have real serial consoles via >>> iLOM, so /v/l/m traffic does bring everything to >>> a standstill. >>> >>> Meanwhile, what's you're opinion about AUTH_FAILED? >>> Should the server return RPCSEC_GSS_CTXPROBLEM >>> in this case instead? If it did, do you think >>> the Linux client would recover by creating a >>> replacement GSS context? >> >> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >> fail with EIO error and will not try to create a new GSS context. So >> yes, I believe it would help if the server returns any of the >> following errors: >> case RPC_AUTH_REJECTEDCRED: >> case RPC_AUTH_REJECTEDVERF: >> case RPCSEC_GSS_CREDPROBLEM: >> case RPCSEC_GSS_CTXPROBLEM: >> >> then the client will recreate the context. > > Also in my testing, I can see that credential cache is per gss flavor. > Just to check, what kernel version is this problem encountered on (I > know you said upstream) but I just want to double check so that I can > look at the correct source code. v4.1.12 (stable) I think. > > Thanks. > >> >>> >>> >>> >>>>>>> —>Andy >>>>>>>> >>>>>>>> One solution is to introduce a quick check before a >>>>>>>> context is used to see if the GSS service bound to it >>>>>>>> matches the GSS service that the caller intends to use. >>>>>>>> I'm not sure how that can be done without exposing a window >>>>>>>> where another caller requests the use of a GSS context and >>>>>>>> grabs the fresh one, before it can be used by our first >>>>>>>> caller and bound to its desired GSS service. >>>>>>>> >>>>>>>> Other solutions might be to somehow isolate the credential >>>>>>>> cache used for lease management operations, or to split >>>>>>>> credential caches by GSS service. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Chuck Lever >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> -- >>>>> Chuck Lever >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> -- >>> Chuck Lever > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-21 21:32 ` Chuck Lever @ 2016-07-25 18:18 ` Olga Kornievskaia 2016-07-29 16:27 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-25 18:18 UTC (permalink / raw) To: Chuck Lever; +Cc: Adamson, Andy, Linux NFS Mailing List On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > > >> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>> >>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>> >>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>> >>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>> >>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>>> <William.Adamson@netapp.com> wrote: >>>>>>>> >>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>> >>>>>>>>> Hi Andy- >>>>>>>>> >>>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>>> report. >>>>>>>>> >>>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>>> and NFSv4.1 mount points. >>>>>>>>> >>>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>>> the network capture. >>>>>>>>> >>>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>>> >>>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>>> lease management operations, which is bound to the client's >>>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>>> Call this GSS context 1. >>>>>>>>> >>>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>>> established for NFS operations associated with that user. >>>>>>>>> Since the reproducer is running as root, this context is >>>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>>> >>>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>>> connection with the server, and performs a RENEW >>>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>>> >>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>>> >>>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>>> problem at all). >>>>>>>>> >>>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>>> service "none" is used for this operation, since it is on >>>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>>> >>>>>>>>> The RPC is accepted, but the server reports >>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>>> >>>>>>>>> The client responds by attempting state recovery. The >>>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>>> credential cache again and finds the recently established >>>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>>> 3 with GSS service "integrity." >>>>>>>>> >>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>>> the client reports that "check lease failed" and >>>>>>>>> terminates state recovery. >>>>>>>>> >>>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>>> stateid with predictable results. The client again tries >>>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>>> >>>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>>> management operations, attempt to use the same GSS context >>>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>>> services. >>>>>>>> >>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>>> >>>>>>>> >>>>>>>>> The key issue seems to be why, when the mount >>>>>>>>> is first established, the client is correctly able to >>>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>>> context with two different GSS services. >>>>>>>> >>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>>> >>>>>>> I agree with Andy. It must be a tight race. >>>>>> >>>>>> In one capture I see something like this after >>>>>> the server restarts: >>>>>> >>>>>> SYN >>>>>> SYN, ACK >>>>>> ACK >>>>>> C WRITE >>>>>> C SEQUENCE >>>>>> R WRITE -> CTX_PROBLEM >>>>>> R SEQUENCE -> CTX_PROBLEM >>>>>> C NULL (KRB5_AP_REQ) >>>>>> R NULL (KRB5_AP_REP) >>>>>> C WRITE >>>>>> C SEQUENCE >>>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>>> R SEQUENCE -> AUTH_FAILED >>>>>> >>>>>> Andy's theory neatly explains this behavior. >>>>>> >>>>>> >>>>>>> I have tried to reproduce >>>>>>> your scenario and in my tests of rebooting the server all recover >>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>>> the new context is established and then RENEW using integrity service >>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>>> >>>>>> If one operation is allowed to complete, then >>>>>> the other will always recognize that another >>>>>> fresh GSS context is needed. If two are sent >>>>>> at the same time, they race and one always >>>>>> fails. >>>>>> >>>>>> Helen's test includes a second idle mount point >>>>>> (sec=krb5i) and maybe that is needed to trigger >>>>>> the race? >>>>> >>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>>> failure (gssd optionally) (i realize that it might alter the timings >>>>> and not hit the issue but worth a shot)? >>>> >>>> I'm sure that's fine. An internal tester hit this, >>>> not a customer, so I will ask. >>>> >>>> I agree, though, that timing might be a problem: >>>> these systems all have real serial consoles via >>>> iLOM, so /v/l/m traffic does bring everything to >>>> a standstill. >>>> >>>> Meanwhile, what's you're opinion about AUTH_FAILED? >>>> Should the server return RPCSEC_GSS_CTXPROBLEM >>>> in this case instead? If it did, do you think >>>> the Linux client would recover by creating a >>>> replacement GSS context? >>> >>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >>> fail with EIO error and will not try to create a new GSS context. So >>> yes, I believe it would help if the server returns any of the >>> following errors: >>> case RPC_AUTH_REJECTEDCRED: >>> case RPC_AUTH_REJECTEDVERF: >>> case RPCSEC_GSS_CREDPROBLEM: >>> case RPCSEC_GSS_CTXPROBLEM: >>> >>> then the client will recreate the context. >> >> Also in my testing, I can see that credential cache is per gss flavor. >> Just to check, what kernel version is this problem encountered on (I >> know you said upstream) but I just want to double check so that I can >> look at the correct source code. > > v4.1.12 (stable) I think. Also, can you share the network trace? > > >> >> Thanks. >> >>> >>>> >>>> >>>> >>>>>>>> —>Andy >>>>>>>>> >>>>>>>>> One solution is to introduce a quick check before a >>>>>>>>> context is used to see if the GSS service bound to it >>>>>>>>> matches the GSS service that the caller intends to use. >>>>>>>>> I'm not sure how that can be done without exposing a window >>>>>>>>> where another caller requests the use of a GSS context and >>>>>>>>> grabs the fresh one, before it can be used by our first >>>>>>>>> caller and bound to its desired GSS service. >>>>>>>>> >>>>>>>>> Other solutions might be to somehow isolate the credential >>>>>>>>> cache used for lease management operations, or to split >>>>>>>>> credential caches by GSS service. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Chuck Lever >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> -- >>>>>> Chuck Lever >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> -- >>>> Chuck Lever >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-25 18:18 ` Olga Kornievskaia @ 2016-07-29 16:27 ` Olga Kornievskaia 2016-07-29 16:38 ` Chuck Lever 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-29 16:27 UTC (permalink / raw) To: Chuck Lever; +Cc: Adamson, Andy, Linux NFS Mailing List On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >> >>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>> >>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>> >>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>> >>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>> >>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>> >>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>>>> <William.Adamson@netapp.com> wrote: >>>>>>>>> >>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>> >>>>>>>>>> Hi Andy- >>>>>>>>>> >>>>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>>>> report. >>>>>>>>>> >>>>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>>>> and NFSv4.1 mount points. >>>>>>>>>> >>>>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>>>> the network capture. >>>>>>>>>> >>>>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>>>> >>>>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>>>> lease management operations, which is bound to the client's >>>>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>>>> Call this GSS context 1. >>>>>>>>>> >>>>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>>>> established for NFS operations associated with that user. >>>>>>>>>> Since the reproducer is running as root, this context is >>>>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>>>> >>>>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>>>> connection with the server, and performs a RENEW >>>>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>>>> >>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>>>> >>>>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>>>> problem at all). >>>>>>>>>> >>>>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>>>> service "none" is used for this operation, since it is on >>>>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>>>> >>>>>>>>>> The RPC is accepted, but the server reports >>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>>>> >>>>>>>>>> The client responds by attempting state recovery. The >>>>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>>>> credential cache again and finds the recently established >>>>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>>>> 3 with GSS service "integrity." >>>>>>>>>> >>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>>>> the client reports that "check lease failed" and >>>>>>>>>> terminates state recovery. >>>>>>>>>> >>>>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>>>> stateid with predictable results. The client again tries >>>>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>>>> >>>>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>>>> management operations, attempt to use the same GSS context >>>>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>>>> services. >>>>>>>>> >>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>>>> >>>>>>>>> >>>>>>>>>> The key issue seems to be why, when the mount >>>>>>>>>> is first established, the client is correctly able to >>>>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>>>> context with two different GSS services. >>>>>>>>> >>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>>>> >>>>>>>> I agree with Andy. It must be a tight race. >>>>>>> >>>>>>> In one capture I see something like this after >>>>>>> the server restarts: >>>>>>> >>>>>>> SYN >>>>>>> SYN, ACK >>>>>>> ACK >>>>>>> C WRITE >>>>>>> C SEQUENCE >>>>>>> R WRITE -> CTX_PROBLEM >>>>>>> R SEQUENCE -> CTX_PROBLEM >>>>>>> C NULL (KRB5_AP_REQ) >>>>>>> R NULL (KRB5_AP_REP) >>>>>>> C WRITE >>>>>>> C SEQUENCE >>>>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>>>> R SEQUENCE -> AUTH_FAILED >>>>>>> >>>>>>> Andy's theory neatly explains this behavior. >>>>>>> >>>>>>> >>>>>>>> I have tried to reproduce >>>>>>>> your scenario and in my tests of rebooting the server all recover >>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>>>> the new context is established and then RENEW using integrity service >>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>>>> >>>>>>> If one operation is allowed to complete, then >>>>>>> the other will always recognize that another >>>>>>> fresh GSS context is needed. If two are sent >>>>>>> at the same time, they race and one always >>>>>>> fails. >>>>>>> >>>>>>> Helen's test includes a second idle mount point >>>>>>> (sec=krb5i) and maybe that is needed to trigger >>>>>>> the race? >>>>>> >>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>>>> failure (gssd optionally) (i realize that it might alter the timings >>>>>> and not hit the issue but worth a shot)? >>>>> >>>>> I'm sure that's fine. An internal tester hit this, >>>>> not a customer, so I will ask. >>>>> >>>>> I agree, though, that timing might be a problem: >>>>> these systems all have real serial consoles via >>>>> iLOM, so /v/l/m traffic does bring everything to >>>>> a standstill. >>>>> >>>>> Meanwhile, what's you're opinion about AUTH_FAILED? >>>>> Should the server return RPCSEC_GSS_CTXPROBLEM >>>>> in this case instead? If it did, do you think >>>>> the Linux client would recover by creating a >>>>> replacement GSS context? >>>> >>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >>>> fail with EIO error and will not try to create a new GSS context. So >>>> yes, I believe it would help if the server returns any of the >>>> following errors: >>>> case RPC_AUTH_REJECTEDCRED: >>>> case RPC_AUTH_REJECTEDVERF: >>>> case RPCSEC_GSS_CREDPROBLEM: >>>> case RPCSEC_GSS_CTXPROBLEM: >>>> >>>> then the client will recreate the context. >>> >>> Also in my testing, I can see that credential cache is per gss flavor. >>> Just to check, what kernel version is this problem encountered on (I >>> know you said upstream) but I just want to double check so that I can >>> look at the correct source code. >> >> v4.1.12 (stable) I think. > > Also, can you share the network trace? Hi Chuck, I was finally able to reproduce the condition you were seeing (i.e., the use of the same context for different gss services). I enabled rpcdebug rpc auth and I can see that the 2nd request ends up finding a gss_upcall message because it's just matched by the uid. There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying that if there is upcall for an uid then it won't add another upcall. So I think the decision is made right there to share the same context no matter the gss service. > >> >> >>> >>> Thanks. >>> >>>> >>>>> >>>>> >>>>> >>>>>>>>> —>Andy >>>>>>>>>> >>>>>>>>>> One solution is to introduce a quick check before a >>>>>>>>>> context is used to see if the GSS service bound to it >>>>>>>>>> matches the GSS service that the caller intends to use. >>>>>>>>>> I'm not sure how that can be done without exposing a window >>>>>>>>>> where another caller requests the use of a GSS context and >>>>>>>>>> grabs the fresh one, before it can be used by our first >>>>>>>>>> caller and bound to its desired GSS service. >>>>>>>>>> >>>>>>>>>> Other solutions might be to somehow isolate the credential >>>>>>>>>> cache used for lease management operations, or to split >>>>>>>>>> credential caches by GSS service. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Chuck Lever >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>>> -- >>>>>>> Chuck Lever >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> -- >>>>> Chuck Lever >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-29 16:27 ` Olga Kornievskaia @ 2016-07-29 16:38 ` Chuck Lever 2016-07-29 17:07 ` Adamson, Andy 2016-08-02 18:06 ` J. Bruce Fields 0 siblings, 2 replies; 25+ messages in thread From: Chuck Lever @ 2016-07-29 16:38 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: Adamson, Andy, Linux NFS Mailing List > On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > > On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>> >>> >>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> >>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>> >>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>> >>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>> >>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>> >>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>>>>> <William.Adamson@netapp.com> wrote: >>>>>>>>>> >>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Andy- >>>>>>>>>>> >>>>>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>>>>> report. >>>>>>>>>>> >>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>>>>> and NFSv4.1 mount points. >>>>>>>>>>> >>>>>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>>>>> the network capture. >>>>>>>>>>> >>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>>>>> >>>>>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>>>>> lease management operations, which is bound to the client's >>>>>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>>>>> Call this GSS context 1. >>>>>>>>>>> >>>>>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>>>>> established for NFS operations associated with that user. >>>>>>>>>>> Since the reproducer is running as root, this context is >>>>>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>>>>> >>>>>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>>>>> connection with the server, and performs a RENEW >>>>>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>>>>> >>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>>>>> >>>>>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>>>>> problem at all). >>>>>>>>>>> >>>>>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>>>>> service "none" is used for this operation, since it is on >>>>>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>>>>> >>>>>>>>>>> The RPC is accepted, but the server reports >>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>>>>> >>>>>>>>>>> The client responds by attempting state recovery. The >>>>>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>>>>> credential cache again and finds the recently established >>>>>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>>>>> 3 with GSS service "integrity." >>>>>>>>>>> >>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>>>>> the client reports that "check lease failed" and >>>>>>>>>>> terminates state recovery. >>>>>>>>>>> >>>>>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>>>>> stateid with predictable results. The client again tries >>>>>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>>>>> >>>>>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>>>>> management operations, attempt to use the same GSS context >>>>>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>>>>> services. >>>>>>>>>> >>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> The key issue seems to be why, when the mount >>>>>>>>>>> is first established, the client is correctly able to >>>>>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>>>>> context with two different GSS services. >>>>>>>>>> >>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>>>>> >>>>>>>>> I agree with Andy. It must be a tight race. >>>>>>>> >>>>>>>> In one capture I see something like this after >>>>>>>> the server restarts: >>>>>>>> >>>>>>>> SYN >>>>>>>> SYN, ACK >>>>>>>> ACK >>>>>>>> C WRITE >>>>>>>> C SEQUENCE >>>>>>>> R WRITE -> CTX_PROBLEM >>>>>>>> R SEQUENCE -> CTX_PROBLEM >>>>>>>> C NULL (KRB5_AP_REQ) >>>>>>>> R NULL (KRB5_AP_REP) >>>>>>>> C WRITE >>>>>>>> C SEQUENCE >>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>>>>> R SEQUENCE -> AUTH_FAILED >>>>>>>> >>>>>>>> Andy's theory neatly explains this behavior. >>>>>>>> >>>>>>>> >>>>>>>>> I have tried to reproduce >>>>>>>>> your scenario and in my tests of rebooting the server all recover >>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>>>>> the new context is established and then RENEW using integrity service >>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>>>>> >>>>>>>> If one operation is allowed to complete, then >>>>>>>> the other will always recognize that another >>>>>>>> fresh GSS context is needed. If two are sent >>>>>>>> at the same time, they race and one always >>>>>>>> fails. >>>>>>>> >>>>>>>> Helen's test includes a second idle mount point >>>>>>>> (sec=krb5i) and maybe that is needed to trigger >>>>>>>> the race? >>>>>>> >>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>>>>> failure (gssd optionally) (i realize that it might alter the timings >>>>>>> and not hit the issue but worth a shot)? >>>>>> >>>>>> I'm sure that's fine. An internal tester hit this, >>>>>> not a customer, so I will ask. >>>>>> >>>>>> I agree, though, that timing might be a problem: >>>>>> these systems all have real serial consoles via >>>>>> iLOM, so /v/l/m traffic does bring everything to >>>>>> a standstill. >>>>>> >>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? >>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM >>>>>> in this case instead? If it did, do you think >>>>>> the Linux client would recover by creating a >>>>>> replacement GSS context? >>>>> >>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >>>>> fail with EIO error and will not try to create a new GSS context. So >>>>> yes, I believe it would help if the server returns any of the >>>>> following errors: >>>>> case RPC_AUTH_REJECTEDCRED: >>>>> case RPC_AUTH_REJECTEDVERF: >>>>> case RPCSEC_GSS_CREDPROBLEM: >>>>> case RPCSEC_GSS_CTXPROBLEM: >>>>> >>>>> then the client will recreate the context. >>>> >>>> Also in my testing, I can see that credential cache is per gss flavor. >>>> Just to check, what kernel version is this problem encountered on (I >>>> know you said upstream) but I just want to double check so that I can >>>> look at the correct source code. >>> >>> v4.1.12 (stable) I think. >> >> Also, can you share the network trace? > > Hi Chuck, > > I was finally able to reproduce the condition you were seeing (i.e., > the use of the same context for different gss services). > > I enabled rpcdebug rpc auth and I can see that the 2nd request ends up > finding a gss_upcall message because it's just matched by the uid. > There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying > that if there is upcall for an uid then it won't add another upcall. > So I think the decision is made right there to share the same context > no matter the gss service. If I understand correctly, that's just what Andy predicted. That check needs to be changed to allow another upcall to be queued if the UID matches but the GSS service does not. -- Chuck Lever ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-29 16:38 ` Chuck Lever @ 2016-07-29 17:07 ` Adamson, Andy 2016-07-29 17:32 ` Adamson, Andy 2016-08-02 18:06 ` J. Bruce Fields 1 sibling, 1 reply; 25+ messages in thread From: Adamson, Andy @ 2016-07-29 17:07 UTC (permalink / raw) To: Chuck Lever; +Cc: Olga Kornievskaia, Adamson, Andy, Linux NFS Mailing List DQo+IE9uIEp1bCAyOSwgMjAxNiwgYXQgMTI6MzggUE0sIENodWNrIExldmVyIDxjaHVjay5sZXZl ckBvcmFjbGUuY29tPiB3cm90ZToNCj4gDQo+PiANCj4+IE9uIEp1bCAyOSwgMjAxNiwgYXQgMTI6 MjcgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+PiANCj4+ IE9uIE1vbiwgSnVsIDI1LCAyMDE2IGF0IDI6MTggUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xv QHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgNTozMiBQTSwg Q2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+PiANCj4+Pj4g DQo+Pj4+PiBPbiBKdWwgMjEsIDIwMTYsIGF0IDEwOjQ2IFBNLCBPbGdhIEtvcm5pZXZza2FpYSA8 YWdsb0B1bWljaC5lZHU+IHdyb3RlOg0KPj4+Pj4gDQo+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIw MTYgYXQgMzo1NCBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3cm90ZToN Cj4+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgMTo1NiBQTSwgQ2h1Y2sgTGV2ZXIgPGNo dWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+PiANCj4+Pj4+Pj4+IE9uIEp1bCAy MSwgMjAxNiwgYXQgNjowNCBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3 cm90ZToNCj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgMjo1NSBB TSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4+ IA0KPj4+Pj4+Pj4+PiBPbiBKdWwgMjAsIDIwMTYsIGF0IDY6NTYgUE0sIE9sZ2EgS29ybmlldnNr YWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBPbiBX ZWQsIEp1bCAyMCwgMjAxNiBhdCA1OjE0IEFNLCBBZGFtc29uLCBBbmR5DQo+Pj4+Pj4+Pj4+IDxX aWxsaWFtLkFkYW1zb25AbmV0YXBwLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+PiBPbiBKdWwgMTksIDIwMTYsIGF0IDEwOjUxIEFNLCBDaHVjayBMZXZlciA8Y2h1Y2subGV2 ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gSGkgQW5k eS0NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBUaGFua3MgZm9yIHRha2luZyB0aGUgdGlt ZSB0byBkaXNjdXNzIHRoaXMgd2l0aCBtZS4gSSd2ZQ0KPj4+Pj4+Pj4+Pj4+IGNvcGllZCBsaW51 eC1uZnMgdG8gbWFrZSB0aGlzIGUtbWFpbCBhbHNvIGFuIHVwc3RyZWFtIGJ1Zw0KPj4+Pj4+Pj4+ Pj4+IHJlcG9ydC4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBBcyB3ZSBzYXcgaW4gdGhl IG5ldHdvcmsgY2FwdHVyZSwgcmVjb3Zlcnkgb2YgR1NTIGNvbnRleHRzDQo+Pj4+Pj4+Pj4+Pj4g YWZ0ZXIgYSBzZXJ2ZXIgcmVib290IGZhaWxzIGluIGNlcnRhaW4gY2FzZXMgd2l0aCBORlN2NC4w DQo+Pj4+Pj4+Pj4+Pj4gYW5kIE5GU3Y0LjEgbW91bnQgcG9pbnRzLg0KPj4+Pj4+Pj4+Pj4+IA0K Pj4+Pj4+Pj4+Pj4+IFRoZSByZXByb2R1Y2VyIGlzIGEgc2ltcGxlIHByb2dyYW0gdGhhdCBnZW5l cmF0ZXMgb25lIE5GUw0KPj4+Pj4+Pj4+Pj4+IFdSSVRFIHBlcmlvZGljYWxseSwgcnVuIHdoaWxl IHRoZSBzZXJ2ZXIgcmVwZWF0ZWRseSByZWJvb3RzDQo+Pj4+Pj4+Pj4+Pj4gKG9yIG9uZSBjbHVz dGVyIGhlYWQgZmFpbHMgb3ZlciB0byB0aGUgb3RoZXIgYW5kIGJhY2spLiBUaGUNCj4+Pj4+Pj4+ Pj4+PiBnb2FsIG9mIHRoZSByZXByb2R1Y2VyIGlzIHRvIGlkZW50aWZ5IHByb2JsZW1zIHdpdGgg c3RhdGUNCj4+Pj4+Pj4+Pj4+PiByZWNvdmVyeSB3aXRob3V0IGEgbG90IG9mIG90aGVyIEkvTyBn b2luZyBvbiB0byBjbHV0dGVyIHVwDQo+Pj4+Pj4+Pj4+Pj4gdGhlIG5ldHdvcmsgY2FwdHVyZS4N Cj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBJbiB0aGUgZmFpbGluZyBjYXNlLCBzZWM9a3Ji NSBpcyBzcGVjaWZpZWQgb24gdGhlIG1vdW50DQo+Pj4+Pj4+Pj4+Pj4gcG9pbnQsIGFuZCB0aGUg cmVwcm9kdWNlciBpcyBydW4gYXMgcm9vdC4gV2UndmUgZm91bmQgdGhpcw0KPj4+Pj4+Pj4+Pj4+ IGNvbWJpbmF0aW9uIGZhaWxzIHdpdGggYm90aCBORlN2NC4wIGFuZCBORlN2NC4xLg0KPj4+Pj4+ Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+IEF0IG1vdW50IHRpbWUsIHRoZSBjbGllbnQgZXN0YWJsaXNo ZXMgYSBHU1MgY29udGV4dCBmb3INCj4+Pj4+Pj4+Pj4+PiBsZWFzZSBtYW5hZ2VtZW50IG9wZXJh dGlvbnMsIHdoaWNoIGlzIGJvdW5kIHRvIHRoZSBjbGllbnQncw0KPj4+Pj4+Pj4+Pj4+IE5GUyBz ZXJ2aWNlIHByaW5jaXBhbCBhbmQgdXNlcyBHU1Mgc2VydmljZSAiaW50ZWdyaXR5LiINCj4+Pj4+ Pj4+Pj4+PiBDYWxsIHRoaXMgR1NTIGNvbnRleHQgMS4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+PiBXaGVuIHRoZSByZXByb2R1Y2VyIHN0YXJ0cywgYSBzZWNvbmQgR1NTIGNvbnRleHQgaXMN Cj4+Pj4+Pj4+Pj4+PiBlc3RhYmxpc2hlZCBmb3IgTkZTIG9wZXJhdGlvbnMgYXNzb2NpYXRlZCB3 aXRoIHRoYXQgdXNlci4NCj4+Pj4+Pj4+Pj4+PiBTaW5jZSB0aGUgcmVwcm9kdWNlciBpcyBydW5u aW5nIGFzIHJvb3QsIHRoaXMgY29udGV4dCBpcw0KPj4+Pj4+Pj4+Pj4+IGFsc28gYm91bmQgdG8g dGhlIGNsaWVudCdzIE5GUyBzZXJ2aWNlIHByaW5jaXBhbCwgYnV0IGl0DQo+Pj4+Pj4+Pj4+Pj4g dXNlcyB0aGUgR1NTIHNlcnZpY2UgIm5vbmUiIChyZWZsZWN0aW5nIHRoZSBleHBsaWNpdA0KPj4+ Pj4+Pj4+Pj4+IHJlcXVlc3QgZm9yICJzZWM9a3JiNSIpLiBDYWxsIHRoaXMgR1NTIGNvbnRleHQg Mi4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBBZnRlciB0aGUgc2VydmVyIHJlYm9vdHMs IHRoZSBjbGllbnQgcmUtZXN0YWJsaXNoZXMgYSBUQ1ANCj4+Pj4+Pj4+Pj4+PiBjb25uZWN0aW9u IHdpdGggdGhlIHNlcnZlciwgYW5kIHBlcmZvcm1zIGEgUkVORVcNCj4+Pj4+Pj4+Pj4+PiBvcGVy YXRpb24gdXNpbmcgY29udGV4dCAxLiBUaGFua3MgdG8gdGhlIHNlcnZlciByZWJvb3QsDQo+Pj4+ Pj4+Pj4+Pj4gY29udGV4dHMgMSBhbmQgMiBhcmUgbm93IHN0YWxlLiBUaGUgc2VydmVyIHRodXMg cmVqZWN0cw0KPj4+Pj4+Pj4+Pj4+IHRoZSBSUEMgd2l0aCBSUENTRUNfR1NTX0NUWFBST0JMRU0u DQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCBwZXJmb3JtcyBhIEdTU19J TklUX1NFQ19DT05URVhUIHZpYSBhbiBORlN2NA0KPj4+Pj4+Pj4+Pj4+IE5VTEwgb3BlcmF0aW9u LiBDYWxsIHRoaXMgR1NTIGNvbnRleHQgMy4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBJ bnRlcmVzdGluZ2x5LCB0aGUgY2xpZW50IGRvZXMgbm90IHJlc2VuZCB0aGUgUkVORVcNCj4+Pj4+ Pj4+Pj4+PiBvcGVyYXRpb24gYXQgdGhpcyBwb2ludCAoaWYgaXQgZGlkLCB3ZSB3b3VsZG4ndCBz ZWUgdGhpcw0KPj4+Pj4+Pj4+Pj4+IHByb2JsZW0gYXQgYWxsKS4NCj4+Pj4+Pj4+Pj4+PiANCj4+ Pj4+Pj4+Pj4+PiBUaGUgY2xpZW50IHRoZW4gYXR0ZW1wdHMgdG8gcmVzdW1lIHRoZSByZXByb2R1 Y2VyIHdvcmtsb2FkLg0KPj4+Pj4+Pj4+Pj4+IEl0IHNlbmRzIGFuIE5GU3Y0IFdSSVRFIG9wZXJh dGlvbiwgdXNpbmcgdGhlIGZpcnN0IGF2YWlsYWJsZQ0KPj4+Pj4+Pj4+Pj4+IEdTUyBjb250ZXh0 IGluIFVJRCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgd2hpY2ggaXMgY29udGV4dCAzLA0KPj4+Pj4+ Pj4+Pj4+IGFscmVhZHkgYm91bmQgdG8gdGhlIGNsaWVudCdzIE5GUyBzZXJ2aWNlIHByaW5jaXBh bC4gQnV0IEdTUw0KPj4+Pj4+Pj4+Pj4+IHNlcnZpY2UgIm5vbmUiIGlzIHVzZWQgZm9yIHRoaXMg b3BlcmF0aW9uLCBzaW5jZSBpdCBpcyBvbg0KPj4+Pj4+Pj4+Pj4+IGJlaGFsZiBvZiB0aGUgbW91 bnQgd2hlcmUgc2VjPWtyYjUgd2FzIHNwZWNpZmllZC4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+PiBUaGUgUlBDIGlzIGFjY2VwdGVkLCBidXQgdGhlIHNlcnZlciByZXBvcnRzDQo+Pj4+Pj4+ Pj4+Pj4gTkZTNEVSUl9TVEFMRV9TVEFURUlELCBzaW5jZSBpdCBoYXMgcmVjZW50bHkgcmVib290 ZWQuDQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCByZXNwb25kcyBieSBh dHRlbXB0aW5nIHN0YXRlIHJlY292ZXJ5LiBUaGUNCj4+Pj4+Pj4+Pj4+PiBmaXJzdCBvcGVyYXRp b24gaXQgdHJpZXMgaXMgYW5vdGhlciBSRU5FVy4gU2luY2UgdGhpcyBpcw0KPj4+Pj4+Pj4+Pj4+ IGEgbGVhc2UgbWFuYWdlbWVudCBvcGVyYXRpb24sIHRoZSBjbGllbnQgbG9va3MgaW4gVUlEIDAn cw0KPj4+Pj4+Pj4+Pj4+IGNyZWRlbnRpYWwgY2FjaGUgYWdhaW4gYW5kIGZpbmRzIHRoZSByZWNl bnRseSBlc3RhYmxpc2hlZA0KPj4+Pj4+Pj4+Pj4+IGNvbnRleHQgMy4gSXQgdHJpZXMgdGhlIFJF TkVXIG9wZXJhdGlvbiB1c2luZyBHU1MgY29udGV4dA0KPj4+Pj4+Pj4+Pj4+IDMgd2l0aCBHU1Mg c2VydmljZSAiaW50ZWdyaXR5LiINCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBUaGUgc2Vy dmVyIHJlamVjdHMgdGhlIFJFTkVXIFJQQyB3aXRoIEFVVEhfRkFJTEVELCBhbmQNCj4+Pj4+Pj4+ Pj4+PiB0aGUgY2xpZW50IHJlcG9ydHMgdGhhdCAiY2hlY2sgbGVhc2UgZmFpbGVkIiBhbmQNCj4+ Pj4+Pj4+Pj4+PiB0ZXJtaW5hdGVzIHN0YXRlIHJlY292ZXJ5Lg0KPj4+Pj4+Pj4+Pj4+IA0KPj4+ Pj4+Pj4+Pj4+IFRoZSBjbGllbnQgcmUtZHJpdmVzIHRoZSBXUklURSBvcGVyYXRpb24gd2l0aCB0 aGUgc3RhbGUNCj4+Pj4+Pj4+Pj4+PiBzdGF0ZWlkIHdpdGggcHJlZGljdGFibGUgcmVzdWx0cy4g VGhlIGNsaWVudCBhZ2FpbiB0cmllcw0KPj4+Pj4+Pj4+Pj4+IHRvIHJlY292ZXIgc3RhdGUgYnkg c2VuZGluZyBhIFJFTkVXLCBhbmQgc3RpbGwgdXNlcyB0aGUNCj4+Pj4+Pj4+Pj4+PiBzYW1lIEdT UyBjb250ZXh0IDMgd2l0aCBzZXJ2aWNlICJpbnRlZ3JpdHkiIGFuZCBnZXRzIHRoZQ0KPj4+Pj4+ Pj4+Pj4+IHNhbWUgcmVzdWx0LiBBIChwZXJoYXBzIHNsb3ctbW90aW9uKSBTVEFMRV9TVEFURUlE IGxvb3ANCj4+Pj4+Pj4+Pj4+PiBlbnN1ZXMsIGFuZCB0aGUgY2xpZW50IG1vdW50IHBvaW50IGlz IGRlYWRsb2NrZWQuDQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gWW91ciBhbmFseXNpcyB3 YXMgdGhhdCBiZWNhdXNlIHRoZSByZXByb2R1Y2VyIGlzIHJ1biBhcw0KPj4+Pj4+Pj4+Pj4+IHJv b3QsIGJvdGggdGhlIHJlcHJvZHVjZXIncyBJL08gb3BlcmF0aW9ucywgYW5kIGxlYXNlDQo+Pj4+ Pj4+Pj4+Pj4gbWFuYWdlbWVudCBvcGVyYXRpb25zLCBhdHRlbXB0IHRvIHVzZSB0aGUgc2FtZSBH U1MgY29udGV4dA0KPj4+Pj4+Pj4+Pj4+IGluIFVJRCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgYnV0 IGVhY2ggdXNlcyBkaWZmZXJlbnQgR1NTDQo+Pj4+Pj4+Pj4+Pj4gc2VydmljZXMuDQo+Pj4+Pj4+ Pj4+PiANCj4+Pj4+Pj4+Pj4+IEFzIFJGQzIyMDMgc3RhdGVzLCAiSW4gYSBjcmVhdGlvbiByZXF1 ZXN0LCB0aGUgc2VxX251bSBhbmQgc2VydmljZSBmaWVsZHMgYXJlIHVuZGVmaW5lZCBhbmQgYm90 aCBtdXN0IGJlIGlnbm9yZWQgYnkgdGhlIHNlcnZlcuKAnQ0KPj4+Pj4+Pj4+Pj4gU28gYSBjb250 ZXh0IGNyZWF0aW9uIHJlcXVlc3Qgd2hpbGUga2lja2VkIG9mZiBieSBhbiBvcGVyYXRpb24gd2l0 aCBhIHNlcnZpY2UgYXR0YWNoZWQgKGUuZy4gV1JJVEUgdXNlcyBycGNfZ3NzX3N2Y19ub25lIGFu ZCBSRU5FVyB1c2VzIHJwY19nc3Nfc3ZjX2ludGVncml0eSksIGNhbiBiZSB1c2VkIGJ5IGVpdGhl ciBzZXJ2aWNlIGxldmVsLg0KPj4+Pj4+Pj4+Pj4gQUZBSUNTIGEgc2luZ2xlIEdTUyBjb250ZXh0 IGNvdWxkIGluIHRoZW9yeSBiZSB1c2VkIGZvciBhbGwgc2VydmljZSBsZXZlbHMsIGJ1dCBpbiBw cmFjdGljZSwgR1NTIGNvbnRleHRzIGFyZSByZXN0cmljdGVkIHRvIGEgc2VydmljZSBsZXZlbCAo YnkgY2xpZW50PyBieSBzZXJ2ZXI/ICkgb25jZSB0aGV5IGFyZSB1c2VkLg0KPj4+Pj4+Pj4+Pj4g DQo+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBUaGUga2V5IGlzc3VlIHNlZW1zIHRvIGJlIHdo eSwgd2hlbiB0aGUgbW91bnQNCj4+Pj4+Pj4+Pj4+PiBpcyBmaXJzdCBlc3RhYmxpc2hlZCwgdGhl IGNsaWVudCBpcyBjb3JyZWN0bHkgYWJsZSB0bw0KPj4+Pj4+Pj4+Pj4+IGVzdGFibGlzaCB0d28g c2VwYXJhdGUgR1NTIGNvbnRleHRzIGZvciBVSUQgMDsgYnV0IGFmdGVyDQo+Pj4+Pj4+Pj4+Pj4g YSBzZXJ2ZXIgcmVib290LCB0aGUgY2xpZW50IGF0dGVtcHRzIHRvIHVzZSB0aGUgc2FtZSBHU1MN Cj4+Pj4+Pj4+Pj4+PiBjb250ZXh0IHdpdGggdHdvIGRpZmZlcmVudCBHU1Mgc2VydmljZXMuDQo+ Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+IEkgc3BlY3VsYXRlIHRoYXQgaXQgaXMgYSByYWNlIGJl dHdlZW4gdGhlIFdSSVRFIGFuZCB0aGUgUkVORVcgdG8gdXNlIHRoZSBzYW1lIG5ld2x5IGNyZWF0 ZWQgR1NTIGNvbnRleHQgdGhhdCBoYXMgbm90IGJlZW4gdXNlZCB5ZXQsIGFuZCBzbyBoYXMgbm8g YXNzaWduZWQgc2VydmljZSBsZXZlbCwgYW5kIHRoZSB0d28gcmVxdWVzdHMgcmFjZSB0byBzZXQg dGhlIHNlcnZpY2UgbGV2ZWwuDQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBJIGFncmVlIHdpdGgg QW5keS4gSXQgbXVzdCBiZSBhIHRpZ2h0IHJhY2UuDQo+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4gSW4g b25lIGNhcHR1cmUgSSBzZWUgc29tZXRoaW5nIGxpa2UgdGhpcyBhZnRlcg0KPj4+Pj4+Pj4+IHRo ZSBzZXJ2ZXIgcmVzdGFydHM6DQo+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4gU1lODQo+Pj4+Pj4+Pj4g U1lOLCBBQ0sNCj4+Pj4+Pj4+PiBBQ0sNCj4+Pj4+Pj4+PiBDIFdSSVRFDQo+Pj4+Pj4+Pj4gQyBT RVFVRU5DRQ0KPj4+Pj4+Pj4+IFIgV1JJVEUgLT4gQ1RYX1BST0JMRU0NCj4+Pj4+Pj4+PiBSIFNF UVVFTkNFIC0+IENUWF9QUk9CTEVNDQo+Pj4+Pj4+Pj4gQyBOVUxMIChLUkI1X0FQX1JFUSkNCj4+ Pj4+Pj4+PiBSIE5VTEwgKEtSQjVfQVBfUkVQKQ0KPj4+Pj4+Pj4+IEMgV1JJVEUNCj4+Pj4+Pj4+ PiBDIFNFUVVFTkNFDQo+Pj4+Pj4+Pj4gUiBXUklURSAtPiBORlM0RVJSX1NUQUxFX1NUQVRFSUQN Cj4+Pj4+Pj4+PiBSIFNFUVVFTkNFIC0+IEFVVEhfRkFJTEVEDQo+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+ Pj4gQW5keSdzIHRoZW9yeSBuZWF0bHkgZXhwbGFpbnMgdGhpcyBiZWhhdmlvci4NCj4+Pj4+Pj4+ PiANCj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4gSSBoYXZlIHRyaWVkIHRvIHJlcHJvZHVjZQ0KPj4+ Pj4+Pj4+PiB5b3VyIHNjZW5hcmlvIGFuZCBpbiBteSB0ZXN0cyBvZiByZWJvb3RpbmcgdGhlIHNl cnZlciBhbGwgcmVjb3Zlcg0KPj4+Pj4+Pj4+PiBjb3JyZWN0bHkuIEluIG15IGNhc2UsIGlmIFJF TkVXIHdhcyB0aGUgb25lIGhpdHRpbmcgdGhlIEFVVEhfRVJSIHRoZW4NCj4+Pj4+Pj4+Pj4gdGhl IG5ldyBjb250ZXh0IGlzIGVzdGFibGlzaGVkIGFuZCB0aGVuIFJFTkVXIHVzaW5nIGludGVncml0 eSBzZXJ2aWNlDQo+Pj4+Pj4+Pj4+IGlzIHJldHJpZWQgd2l0aCB0aGUgbmV3IGNvbnRleHQgd2hp Y2ggZ2V0cyBFUlJfU1RBTEVfQ0xJRU5USUQgd2hpY2gNCj4+Pj4+Pj4+Pj4gdGhlbiBjbGllbnQg cmVjb3ZlcnMgZnJvbS4gSWYgaXQncyBhbiBvcGVyYXRpb24gKEkgaGF2ZSBhIEdFVEFUVFIpDQo+ Pj4+Pj4+Pj4+IHRoYXQgZ2V0cyBBVVRIX0VSUiwgdGhlbiBpdCBnZXRzIG5ldyBjb250ZXh0IGFu ZCBpcyByZXRyaWVkIHVzaW5nIG5vbmUNCj4+Pj4+Pj4+Pj4gc2VydmljZS4gVGhlbiBSRU5FVyBn ZXRzIGl0cyBvd24gQVVUSF9FUlIgYXMgaXQgdXNlcyBhIGRpZmZlcmVudA0KPj4+Pj4+Pj4+PiBj b250ZXh0LCBhIG5ldyBjb250ZXh0IGlzIGdvdHRlbiwgUkVORVcgaXMgcmV0cmllZCBvdmVyIGlu dGVncml0eSBhbmQNCj4+Pj4+Pj4+Pj4gZ2V0cyBFUlJfU1RBTEVfQ0xJRU5USUQgd2hpY2ggaXQg cmVjb3ZlcnMgZnJvbS4NCj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+PiBJZiBvbmUgb3BlcmF0aW9uIGlz IGFsbG93ZWQgdG8gY29tcGxldGUsIHRoZW4NCj4+Pj4+Pj4+PiB0aGUgb3RoZXIgd2lsbCBhbHdh eXMgcmVjb2duaXplIHRoYXQgYW5vdGhlcg0KPj4+Pj4+Pj4+IGZyZXNoIEdTUyBjb250ZXh0IGlz IG5lZWRlZC4gSWYgdHdvIGFyZSBzZW50DQo+Pj4+Pj4+Pj4gYXQgdGhlIHNhbWUgdGltZSwgdGhl eSByYWNlIGFuZCBvbmUgYWx3YXlzDQo+Pj4+Pj4+Pj4gZmFpbHMuDQo+Pj4+Pj4+Pj4gDQo+Pj4+ Pj4+Pj4gSGVsZW4ncyB0ZXN0IGluY2x1ZGVzIGEgc2Vjb25kIGlkbGUgbW91bnQgcG9pbnQNCj4+ Pj4+Pj4+PiAoc2VjPWtyYjVpKSBhbmQgbWF5YmUgdGhhdCBpcyBuZWVkZWQgdG8gdHJpZ2dlcg0K Pj4+Pj4+Pj4+IHRoZSByYWNlPw0KPj4+Pj4+Pj4gDQo+Pj4+Pj4+PiBDaHVjaywgYW55IGNoYW5j ZSB0byBnZXQgInJwY2RlYnVnIC1tIHJwYyBhdXRoIiBvdXRwdXQgZHVyaW5nIHRoZQ0KPj4+Pj4+ Pj4gZmFpbHVyZSAoZ3NzZCBvcHRpb25hbGx5KSAoaSByZWFsaXplIHRoYXQgaXQgbWlnaHQgYWx0 ZXIgdGhlIHRpbWluZ3MNCj4+Pj4+Pj4+IGFuZCBub3QgaGl0IHRoZSBpc3N1ZSBidXQgd29ydGgg YSBzaG90KT8NCj4+Pj4+Pj4gDQo+Pj4+Pj4+IEknbSBzdXJlIHRoYXQncyBmaW5lLiBBbiBpbnRl cm5hbCB0ZXN0ZXIgaGl0IHRoaXMsDQo+Pj4+Pj4+IG5vdCBhIGN1c3RvbWVyLCBzbyBJIHdpbGwg YXNrLg0KPj4+Pj4+PiANCj4+Pj4+Pj4gSSBhZ3JlZSwgdGhvdWdoLCB0aGF0IHRpbWluZyBtaWdo dCBiZSBhIHByb2JsZW06DQo+Pj4+Pj4+IHRoZXNlIHN5c3RlbXMgYWxsIGhhdmUgcmVhbCBzZXJp YWwgY29uc29sZXMgdmlhDQo+Pj4+Pj4+IGlMT00sIHNvIC92L2wvbSB0cmFmZmljIGRvZXMgYnJp bmcgZXZlcnl0aGluZyB0bw0KPj4+Pj4+PiBhIHN0YW5kc3RpbGwuDQo+Pj4+Pj4+IA0KPj4+Pj4+ PiBNZWFud2hpbGUsIHdoYXQncyB5b3UncmUgb3BpbmlvbiBhYm91dCBBVVRIX0ZBSUxFRD8NCj4+ Pj4+Pj4gU2hvdWxkIHRoZSBzZXJ2ZXIgcmV0dXJuIFJQQ1NFQ19HU1NfQ1RYUFJPQkxFTQ0KPj4+ Pj4+PiBpbiB0aGlzIGNhc2UgaW5zdGVhZD8gSWYgaXQgZGlkLCBkbyB5b3UgdGhpbmsNCj4+Pj4+ Pj4gdGhlIExpbnV4IGNsaWVudCB3b3VsZCByZWNvdmVyIGJ5IGNyZWF0aW5nIGENCj4+Pj4+Pj4g cmVwbGFjZW1lbnQgR1NTIGNvbnRleHQ/DQo+Pj4+Pj4gDQo+Pj4+Pj4gQWgsIHllcywgSSBlcXVh dGVkIEFVVEhfRkFJTEVEIEFuZCBBVVRIX0VSUk9SIGluIG15IG1pbmQuIElmIGNsaWVudA0KPj4+ Pj4+IHJlY2VpdmVzIHRoZSByZWFzb24gYXMgQVVUSF9GQUlMRUQgYXMgb3Bwb3NlIHRvIENUWFBS T0JMRU0gaXQgd2lsbA0KPj4+Pj4+IGZhaWwgd2l0aCBFSU8gZXJyb3IgYW5kIHdpbGwgbm90IHRy eSB0byBjcmVhdGUgYSBuZXcgR1NTIGNvbnRleHQuIFNvDQo+Pj4+Pj4geWVzLCBJIGJlbGlldmUg aXQgd291bGQgaGVscCBpZiB0aGUgc2VydmVyIHJldHVybnMgYW55IG9mIHRoZQ0KPj4+Pj4+IGZv bGxvd2luZyBlcnJvcnM6DQo+Pj4+Pj4gICAgICAgICAgICAgIGNhc2UgUlBDX0FVVEhfUkVKRUNU RURDUkVEOg0KPj4+Pj4+ICAgICAgICAgICAgICBjYXNlIFJQQ19BVVRIX1JFSkVDVEVEVkVSRjoN Cj4+Pj4+PiAgICAgICAgICAgICAgY2FzZSBSUENTRUNfR1NTX0NSRURQUk9CTEVNOg0KPj4+Pj4+ ICAgICAgICAgICAgICBjYXNlIFJQQ1NFQ19HU1NfQ1RYUFJPQkxFTToNCj4+Pj4+PiANCj4+Pj4+ PiB0aGVuIHRoZSBjbGllbnQgd2lsbCByZWNyZWF0ZSB0aGUgY29udGV4dC4NCj4+Pj4+IA0KPj4+ Pj4gQWxzbyBpbiBteSB0ZXN0aW5nLCBJIGNhbiBzZWUgdGhhdCBjcmVkZW50aWFsIGNhY2hlIGlz IHBlciBnc3MgZmxhdm9yLg0KPj4+Pj4gSnVzdCB0byBjaGVjaywgd2hhdCBrZXJuZWwgdmVyc2lv biBpcyB0aGlzIHByb2JsZW0gZW5jb3VudGVyZWQgb24gKEkNCj4+Pj4+IGtub3cgeW91IHNhaWQg dXBzdHJlYW0pIGJ1dCBJIGp1c3Qgd2FudCB0byBkb3VibGUgY2hlY2sgc28gdGhhdCBJIGNhbg0K Pj4+Pj4gbG9vayBhdCB0aGUgY29ycmVjdCBzb3VyY2UgY29kZS4NCj4+Pj4gDQo+Pj4+IHY0LjEu MTIgKHN0YWJsZSkgSSB0aGluay4NCj4+PiANCj4+PiBBbHNvLCBjYW4geW91IHNoYXJlIHRoZSBu ZXR3b3JrIHRyYWNlPw0KPj4gDQo+PiBIaSBDaHVjaywNCj4+IA0KPj4gSSB3YXMgZmluYWxseSBh YmxlIHRvIHJlcHJvZHVjZSB0aGUgY29uZGl0aW9uIHlvdSB3ZXJlIHNlZWluZyAoaS5lLiwNCj4+ IHRoZSB1c2Ugb2YgdGhlIHNhbWUgY29udGV4dCBmb3IgZGlmZmVyZW50IGdzcyBzZXJ2aWNlcyku DQo+PiANCj4+IEkgZW5hYmxlZCBycGNkZWJ1ZyBycGMgYXV0aCBhbmQgSSBjYW4gc2VlIHRoYXQg dGhlIDJuZCByZXF1ZXN0IGVuZHMgdXANCj4+IGZpbmRpbmcgYSBnc3NfdXBjYWxsIG1lc3NhZ2Ug YmVjYXVzZSBpdCdzIGp1c3QgbWF0Y2hlZCBieSB0aGUgdWlkLg0KPj4gVGhlcmUgaXMgZXZlbiBh IGNvbW1lbnQgaW4gYXV0aF9nc3MvYXV0aF9nc3MuYyBpbiBnc3NfYWRkX21zZygpIHNheWluZw0K Pj4gdGhhdCBpZiB0aGVyZSBpcyB1cGNhbGwgZm9yIGFuIHVpZCB0aGVuIGl0IHdvbid0IGFkZCBh bm90aGVyIHVwY2FsbC4NCj4+IFNvIEkgdGhpbmsgdGhlIGRlY2lzaW9uIGlzIG1hZGUgcmlnaHQg dGhlcmUgdG8gc2hhcmUgdGhlIHNhbWUgY29udGV4dA0KPj4gbm8gbWF0dGVyIHRoZSBnc3Mgc2Vy dmljZS4NCj4gDQo+IElmIEkgdW5kZXJzdGFuZCBjb3JyZWN0bHksIHRoYXQncyBqdXN0IHdoYXQg QW5keSBwcmVkaWN0ZWQuDQo+IA0KPiBUaGF0IGNoZWNrIG5lZWRzIHRvIGJlIGNoYW5nZWQgdG8g YWxsb3cgYW5vdGhlciB1cGNhbGwgdG8gYmUNCj4gcXVldWVkIGlmIHRoZSBVSUQgbWF0Y2hlcyBi dXQgdGhlIEdTUyBzZXJ2aWNlIGRvZXMgbm90Lg0KDQpZZXMsIHdlIG5lZWQgdG8gZW5zdXJlIHRo YXQgdGhlIHNlcnZpY2UgaXMgc2V0IGV2ZW4gdGhvdWdoIGl0IGlzIGlnbm9yZWQgYnkgdGhlIHNl cnZlci4NCg0K4oCUPkFuZHkNCg0KPiANCj4gDQo+IC0tDQo+IENodWNrIExldmVyDQoNCg== ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-29 17:07 ` Adamson, Andy @ 2016-07-29 17:32 ` Adamson, Andy 2016-07-29 22:24 ` Olga Kornievskaia 0 siblings, 1 reply; 25+ messages in thread From: Adamson, Andy @ 2016-07-29 17:32 UTC (permalink / raw) To: Adamson, Andy; +Cc: Chuck Lever, Olga Kornievskaia, Linux NFS Mailing List DQo+IE9uIEp1bCAyOSwgMjAxNiwgYXQgMTowNyBQTSwgQWRhbXNvbiwgQW5keSA8V2lsbGlhbS5B ZGFtc29uQG5ldGFwcC5jb20+IHdyb3RlOg0KPiANCj4+IA0KPj4gT24gSnVsIDI5LCAyMDE2LCBh dCAxMjozOCBQTSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0K Pj4gDQo+Pj4gDQo+Pj4gT24gSnVsIDI5LCAyMDE2LCBhdCAxMjoyNyBQTSwgT2xnYSBLb3JuaWV2 c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3cm90ZToNCj4+PiANCj4+PiBPbiBNb24sIEp1bCAyNSwg MjAxNiBhdCAyOjE4IFBNLCBPbGdhIEtvcm5pZXZza2FpYSA8YWdsb0B1bWljaC5lZHU+IHdyb3Rl Og0KPj4+PiBPbiBUaHUsIEp1bCAyMSwgMjAxNiBhdCA1OjMyIFBNLCBDaHVjayBMZXZlciA8Y2h1 Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+PiANCj4+Pj4+IA0KPj4+Pj4+IE9uIEp1 bCAyMSwgMjAxNiwgYXQgMTA6NDYgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVk dT4gd3JvdGU6DQo+Pj4+Pj4gDQo+Pj4+Pj4+IE9uIFRodSwgSnVsIDIxLCAyMDE2IGF0IDM6NTQg UE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+PiBP biBUaHUsIEp1bCAyMSwgMjAxNiBhdCAxOjU2IFBNLCBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJA b3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+PiBPbiBKdWwgMjEsIDIwMTYs IGF0IDY6MDQgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+ Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgMjo1NSBBTSwgQ2h1 Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4+PiANCj4+ Pj4+Pj4+Pj4+IE9uIEp1bCAyMCwgMjAxNiwgYXQgNjo1NiBQTSwgT2xnYSBLb3JuaWV2c2thaWEg PGFnbG9AdW1pY2guZWR1PiB3cm90ZToNCj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4gT24gV2Vk LCBKdWwgMjAsIDIwMTYgYXQgNToxNCBBTSwgQWRhbXNvbiwgQW5keQ0KPj4+Pj4+Pj4+Pj4gPFdp bGxpYW0uQWRhbXNvbkBuZXRhcHAuY29tPiB3cm90ZToNCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+Pj4gT24gSnVsIDE5LCAyMDE2LCBhdCAxMDo1MSBBTSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxl dmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gSGkg QW5keS0NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IFRoYW5rcyBmb3IgdGFraW5nIHRo ZSB0aW1lIHRvIGRpc2N1c3MgdGhpcyB3aXRoIG1lLiBJJ3ZlDQo+Pj4+Pj4+Pj4+Pj4+IGNvcGll ZCBsaW51eC1uZnMgdG8gbWFrZSB0aGlzIGUtbWFpbCBhbHNvIGFuIHVwc3RyZWFtIGJ1Zw0KPj4+ Pj4+Pj4+Pj4+PiByZXBvcnQuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBBcyB3ZSBz YXcgaW4gdGhlIG5ldHdvcmsgY2FwdHVyZSwgcmVjb3Zlcnkgb2YgR1NTIGNvbnRleHRzDQo+Pj4+ Pj4+Pj4+Pj4+IGFmdGVyIGEgc2VydmVyIHJlYm9vdCBmYWlscyBpbiBjZXJ0YWluIGNhc2VzIHdp dGggTkZTdjQuMA0KPj4+Pj4+Pj4+Pj4+PiBhbmQgTkZTdjQuMSBtb3VudCBwb2ludHMuDQo+Pj4+ Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBUaGUgcmVwcm9kdWNlciBpcyBhIHNpbXBsZSBwcm9n cmFtIHRoYXQgZ2VuZXJhdGVzIG9uZSBORlMNCj4+Pj4+Pj4+Pj4+Pj4gV1JJVEUgcGVyaW9kaWNh bGx5LCBydW4gd2hpbGUgdGhlIHNlcnZlciByZXBlYXRlZGx5IHJlYm9vdHMNCj4+Pj4+Pj4+Pj4+ Pj4gKG9yIG9uZSBjbHVzdGVyIGhlYWQgZmFpbHMgb3ZlciB0byB0aGUgb3RoZXIgYW5kIGJhY2sp LiBUaGUNCj4+Pj4+Pj4+Pj4+Pj4gZ29hbCBvZiB0aGUgcmVwcm9kdWNlciBpcyB0byBpZGVudGlm eSBwcm9ibGVtcyB3aXRoIHN0YXRlDQo+Pj4+Pj4+Pj4+Pj4+IHJlY292ZXJ5IHdpdGhvdXQgYSBs b3Qgb2Ygb3RoZXIgSS9PIGdvaW5nIG9uIHRvIGNsdXR0ZXIgdXANCj4+Pj4+Pj4+Pj4+Pj4gdGhl IG5ldHdvcmsgY2FwdHVyZS4NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IEluIHRoZSBm YWlsaW5nIGNhc2UsIHNlYz1rcmI1IGlzIHNwZWNpZmllZCBvbiB0aGUgbW91bnQNCj4+Pj4+Pj4+ Pj4+Pj4gcG9pbnQsIGFuZCB0aGUgcmVwcm9kdWNlciBpcyBydW4gYXMgcm9vdC4gV2UndmUgZm91 bmQgdGhpcw0KPj4+Pj4+Pj4+Pj4+PiBjb21iaW5hdGlvbiBmYWlscyB3aXRoIGJvdGggTkZTdjQu MCBhbmQgTkZTdjQuMS4NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IEF0IG1vdW50IHRp bWUsIHRoZSBjbGllbnQgZXN0YWJsaXNoZXMgYSBHU1MgY29udGV4dCBmb3INCj4+Pj4+Pj4+Pj4+ Pj4gbGVhc2UgbWFuYWdlbWVudCBvcGVyYXRpb25zLCB3aGljaCBpcyBib3VuZCB0byB0aGUgY2xp ZW50J3MNCj4+Pj4+Pj4+Pj4+Pj4gTkZTIHNlcnZpY2UgcHJpbmNpcGFsIGFuZCB1c2VzIEdTUyBz ZXJ2aWNlICJpbnRlZ3JpdHkuIg0KPj4+Pj4+Pj4+Pj4+PiBDYWxsIHRoaXMgR1NTIGNvbnRleHQg MS4NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IFdoZW4gdGhlIHJlcHJvZHVjZXIgc3Rh cnRzLCBhIHNlY29uZCBHU1MgY29udGV4dCBpcw0KPj4+Pj4+Pj4+Pj4+PiBlc3RhYmxpc2hlZCBm b3IgTkZTIG9wZXJhdGlvbnMgYXNzb2NpYXRlZCB3aXRoIHRoYXQgdXNlci4NCj4+Pj4+Pj4+Pj4+ Pj4gU2luY2UgdGhlIHJlcHJvZHVjZXIgaXMgcnVubmluZyBhcyByb290LCB0aGlzIGNvbnRleHQg aXMNCj4+Pj4+Pj4+Pj4+Pj4gYWxzbyBib3VuZCB0byB0aGUgY2xpZW50J3MgTkZTIHNlcnZpY2Ug cHJpbmNpcGFsLCBidXQgaXQNCj4+Pj4+Pj4+Pj4+Pj4gdXNlcyB0aGUgR1NTIHNlcnZpY2UgIm5v bmUiIChyZWZsZWN0aW5nIHRoZSBleHBsaWNpdA0KPj4+Pj4+Pj4+Pj4+PiByZXF1ZXN0IGZvciAi c2VjPWtyYjUiKS4gQ2FsbCB0aGlzIEdTUyBjb250ZXh0IDIuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+ Pj4+Pj4+Pj4+PiBBZnRlciB0aGUgc2VydmVyIHJlYm9vdHMsIHRoZSBjbGllbnQgcmUtZXN0YWJs aXNoZXMgYSBUQ1ANCj4+Pj4+Pj4+Pj4+Pj4gY29ubmVjdGlvbiB3aXRoIHRoZSBzZXJ2ZXIsIGFu ZCBwZXJmb3JtcyBhIFJFTkVXDQo+Pj4+Pj4+Pj4+Pj4+IG9wZXJhdGlvbiB1c2luZyBjb250ZXh0 IDEuIFRoYW5rcyB0byB0aGUgc2VydmVyIHJlYm9vdCwNCj4+Pj4+Pj4+Pj4+Pj4gY29udGV4dHMg MSBhbmQgMiBhcmUgbm93IHN0YWxlLiBUaGUgc2VydmVyIHRodXMgcmVqZWN0cw0KPj4+Pj4+Pj4+ Pj4+PiB0aGUgUlBDIHdpdGggUlBDU0VDX0dTU19DVFhQUk9CTEVNLg0KPj4+Pj4+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCBwZXJmb3JtcyBhIEdTU19JTklUX1NFQ19DT05URVhU IHZpYSBhbiBORlN2NA0KPj4+Pj4+Pj4+Pj4+PiBOVUxMIG9wZXJhdGlvbi4gQ2FsbCB0aGlzIEdT UyBjb250ZXh0IDMuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBJbnRlcmVzdGluZ2x5 LCB0aGUgY2xpZW50IGRvZXMgbm90IHJlc2VuZCB0aGUgUkVORVcNCj4+Pj4+Pj4+Pj4+Pj4gb3Bl cmF0aW9uIGF0IHRoaXMgcG9pbnQgKGlmIGl0IGRpZCwgd2Ugd291bGRuJ3Qgc2VlIHRoaXMNCj4+ Pj4+Pj4+Pj4+Pj4gcHJvYmxlbSBhdCBhbGwpLg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ Pj4gVGhlIGNsaWVudCB0aGVuIGF0dGVtcHRzIHRvIHJlc3VtZSB0aGUgcmVwcm9kdWNlciB3b3Jr bG9hZC4NCj4+Pj4+Pj4+Pj4+Pj4gSXQgc2VuZHMgYW4gTkZTdjQgV1JJVEUgb3BlcmF0aW9uLCB1 c2luZyB0aGUgZmlyc3QgYXZhaWxhYmxlDQo+Pj4+Pj4+Pj4+Pj4+IEdTUyBjb250ZXh0IGluIFVJ RCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgd2hpY2ggaXMgY29udGV4dCAzLA0KPj4+Pj4+Pj4+Pj4+ PiBhbHJlYWR5IGJvdW5kIHRvIHRoZSBjbGllbnQncyBORlMgc2VydmljZSBwcmluY2lwYWwuIEJ1 dCBHU1MNCj4+Pj4+Pj4+Pj4+Pj4gc2VydmljZSAibm9uZSIgaXMgdXNlZCBmb3IgdGhpcyBvcGVy YXRpb24sIHNpbmNlIGl0IGlzIG9uDQo+Pj4+Pj4+Pj4+Pj4+IGJlaGFsZiBvZiB0aGUgbW91bnQg d2hlcmUgc2VjPWtyYjUgd2FzIHNwZWNpZmllZC4NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+ Pj4+IFRoZSBSUEMgaXMgYWNjZXB0ZWQsIGJ1dCB0aGUgc2VydmVyIHJlcG9ydHMNCj4+Pj4+Pj4+ Pj4+Pj4gTkZTNEVSUl9TVEFMRV9TVEFURUlELCBzaW5jZSBpdCBoYXMgcmVjZW50bHkgcmVib290 ZWQuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBUaGUgY2xpZW50IHJlc3BvbmRzIGJ5 IGF0dGVtcHRpbmcgc3RhdGUgcmVjb3ZlcnkuIFRoZQ0KPj4+Pj4+Pj4+Pj4+PiBmaXJzdCBvcGVy YXRpb24gaXQgdHJpZXMgaXMgYW5vdGhlciBSRU5FVy4gU2luY2UgdGhpcyBpcw0KPj4+Pj4+Pj4+ Pj4+PiBhIGxlYXNlIG1hbmFnZW1lbnQgb3BlcmF0aW9uLCB0aGUgY2xpZW50IGxvb2tzIGluIFVJ RCAwJ3MNCj4+Pj4+Pj4+Pj4+Pj4gY3JlZGVudGlhbCBjYWNoZSBhZ2FpbiBhbmQgZmluZHMgdGhl IHJlY2VudGx5IGVzdGFibGlzaGVkDQo+Pj4+Pj4+Pj4+Pj4+IGNvbnRleHQgMy4gSXQgdHJpZXMg dGhlIFJFTkVXIG9wZXJhdGlvbiB1c2luZyBHU1MgY29udGV4dA0KPj4+Pj4+Pj4+Pj4+PiAzIHdp dGggR1NTIHNlcnZpY2UgImludGVncml0eS4iDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+ PiBUaGUgc2VydmVyIHJlamVjdHMgdGhlIFJFTkVXIFJQQyB3aXRoIEFVVEhfRkFJTEVELCBhbmQN Cj4+Pj4+Pj4+Pj4+Pj4gdGhlIGNsaWVudCByZXBvcnRzIHRoYXQgImNoZWNrIGxlYXNlIGZhaWxl ZCIgYW5kDQo+Pj4+Pj4+Pj4+Pj4+IHRlcm1pbmF0ZXMgc3RhdGUgcmVjb3ZlcnkuDQo+Pj4+Pj4+ Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBUaGUgY2xpZW50IHJlLWRyaXZlcyB0aGUgV1JJVEUgb3Bl cmF0aW9uIHdpdGggdGhlIHN0YWxlDQo+Pj4+Pj4+Pj4+Pj4+IHN0YXRlaWQgd2l0aCBwcmVkaWN0 YWJsZSByZXN1bHRzLiBUaGUgY2xpZW50IGFnYWluIHRyaWVzDQo+Pj4+Pj4+Pj4+Pj4+IHRvIHJl Y292ZXIgc3RhdGUgYnkgc2VuZGluZyBhIFJFTkVXLCBhbmQgc3RpbGwgdXNlcyB0aGUNCj4+Pj4+ Pj4+Pj4+Pj4gc2FtZSBHU1MgY29udGV4dCAzIHdpdGggc2VydmljZSAiaW50ZWdyaXR5IiBhbmQg Z2V0cyB0aGUNCj4+Pj4+Pj4+Pj4+Pj4gc2FtZSByZXN1bHQuIEEgKHBlcmhhcHMgc2xvdy1tb3Rp b24pIFNUQUxFX1NUQVRFSUQgbG9vcA0KPj4+Pj4+Pj4+Pj4+PiBlbnN1ZXMsIGFuZCB0aGUgY2xp ZW50IG1vdW50IHBvaW50IGlzIGRlYWRsb2NrZWQuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+PiBZb3VyIGFuYWx5c2lzIHdhcyB0aGF0IGJlY2F1c2UgdGhlIHJlcHJvZHVjZXIgaXMgcnVu IGFzDQo+Pj4+Pj4+Pj4+Pj4+IHJvb3QsIGJvdGggdGhlIHJlcHJvZHVjZXIncyBJL08gb3BlcmF0 aW9ucywgYW5kIGxlYXNlDQo+Pj4+Pj4+Pj4+Pj4+IG1hbmFnZW1lbnQgb3BlcmF0aW9ucywgYXR0 ZW1wdCB0byB1c2UgdGhlIHNhbWUgR1NTIGNvbnRleHQNCj4+Pj4+Pj4+Pj4+Pj4gaW4gVUlEIDAn cyBjcmVkZW50aWFsIGNhY2hlLCBidXQgZWFjaCB1c2VzIGRpZmZlcmVudCBHU1MNCj4+Pj4+Pj4+ Pj4+Pj4gc2VydmljZXMuDQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gQXMgUkZDMjIwMyBz dGF0ZXMsICJJbiBhIGNyZWF0aW9uIHJlcXVlc3QsIHRoZSBzZXFfbnVtIGFuZCBzZXJ2aWNlIGZp ZWxkcyBhcmUgdW5kZWZpbmVkIGFuZCBib3RoIG11c3QgYmUgaWdub3JlZCBieSB0aGUgc2VydmVy 4oCdDQo+Pj4+Pj4+Pj4+Pj4gU28gYSBjb250ZXh0IGNyZWF0aW9uIHJlcXVlc3Qgd2hpbGUga2lj a2VkIG9mZiBieSBhbiBvcGVyYXRpb24gd2l0aCBhIHNlcnZpY2UgYXR0YWNoZWQgKGUuZy4gV1JJ VEUgdXNlcyBycGNfZ3NzX3N2Y19ub25lIGFuZCBSRU5FVyB1c2VzIHJwY19nc3Nfc3ZjX2ludGVn cml0eSksIGNhbiBiZSB1c2VkIGJ5IGVpdGhlciBzZXJ2aWNlIGxldmVsLg0KPj4+Pj4+Pj4+Pj4+ IEFGQUlDUyBhIHNpbmdsZSBHU1MgY29udGV4dCBjb3VsZCBpbiB0aGVvcnkgYmUgdXNlZCBmb3Ig YWxsIHNlcnZpY2UgbGV2ZWxzLCBidXQgaW4gcHJhY3RpY2UsIEdTUyBjb250ZXh0cyBhcmUgcmVz dHJpY3RlZCB0byBhIHNlcnZpY2UgbGV2ZWwgKGJ5IGNsaWVudD8gYnkgc2VydmVyPyApIG9uY2Ug dGhleSBhcmUgdXNlZC4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ Pj4gVGhlIGtleSBpc3N1ZSBzZWVtcyB0byBiZSB3aHksIHdoZW4gdGhlIG1vdW50DQo+Pj4+Pj4+ Pj4+Pj4+IGlzIGZpcnN0IGVzdGFibGlzaGVkLCB0aGUgY2xpZW50IGlzIGNvcnJlY3RseSBhYmxl IHRvDQo+Pj4+Pj4+Pj4+Pj4+IGVzdGFibGlzaCB0d28gc2VwYXJhdGUgR1NTIGNvbnRleHRzIGZv ciBVSUQgMDsgYnV0IGFmdGVyDQo+Pj4+Pj4+Pj4+Pj4+IGEgc2VydmVyIHJlYm9vdCwgdGhlIGNs aWVudCBhdHRlbXB0cyB0byB1c2UgdGhlIHNhbWUgR1NTDQo+Pj4+Pj4+Pj4+Pj4+IGNvbnRleHQg d2l0aCB0d28gZGlmZmVyZW50IEdTUyBzZXJ2aWNlcy4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+PiBJIHNwZWN1bGF0ZSB0aGF0IGl0IGlzIGEgcmFjZSBiZXR3ZWVuIHRoZSBXUklURSBhbmQg dGhlIFJFTkVXIHRvIHVzZSB0aGUgc2FtZSBuZXdseSBjcmVhdGVkIEdTUyBjb250ZXh0IHRoYXQg aGFzIG5vdCBiZWVuIHVzZWQgeWV0LCBhbmQgc28gaGFzIG5vIGFzc2lnbmVkIHNlcnZpY2UgbGV2 ZWwsIGFuZCB0aGUgdHdvIHJlcXVlc3RzIHJhY2UgdG8gc2V0IHRoZSBzZXJ2aWNlIGxldmVsLg0K Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+PiBJIGFncmVlIHdpdGggQW5keS4gSXQgbXVzdCBiZSBh IHRpZ2h0IHJhY2UuDQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBJbiBvbmUgY2FwdHVyZSBJIHNl ZSBzb21ldGhpbmcgbGlrZSB0aGlzIGFmdGVyDQo+Pj4+Pj4+Pj4+IHRoZSBzZXJ2ZXIgcmVzdGFy dHM6DQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBTWU4NCj4+Pj4+Pj4+Pj4gU1lOLCBBQ0sNCj4+ Pj4+Pj4+Pj4gQUNLDQo+Pj4+Pj4+Pj4+IEMgV1JJVEUNCj4+Pj4+Pj4+Pj4gQyBTRVFVRU5DRQ0K Pj4+Pj4+Pj4+PiBSIFdSSVRFIC0+IENUWF9QUk9CTEVNDQo+Pj4+Pj4+Pj4+IFIgU0VRVUVOQ0Ug LT4gQ1RYX1BST0JMRU0NCj4+Pj4+Pj4+Pj4gQyBOVUxMIChLUkI1X0FQX1JFUSkNCj4+Pj4+Pj4+ Pj4gUiBOVUxMIChLUkI1X0FQX1JFUCkNCj4+Pj4+Pj4+Pj4gQyBXUklURQ0KPj4+Pj4+Pj4+PiBD IFNFUVVFTkNFDQo+Pj4+Pj4+Pj4+IFIgV1JJVEUgLT4gTkZTNEVSUl9TVEFMRV9TVEFURUlEDQo+ Pj4+Pj4+Pj4+IFIgU0VRVUVOQ0UgLT4gQVVUSF9GQUlMRUQNCj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+ Pj4+IEFuZHkncyB0aGVvcnkgbmVhdGx5IGV4cGxhaW5zIHRoaXMgYmVoYXZpb3IuDQo+Pj4+Pj4+ Pj4+IA0KPj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+IEkgaGF2ZSB0cmllZCB0byByZXByb2R1Y2UN Cj4+Pj4+Pj4+Pj4+IHlvdXIgc2NlbmFyaW8gYW5kIGluIG15IHRlc3RzIG9mIHJlYm9vdGluZyB0 aGUgc2VydmVyIGFsbCByZWNvdmVyDQo+Pj4+Pj4+Pj4+PiBjb3JyZWN0bHkuIEluIG15IGNhc2Us IGlmIFJFTkVXIHdhcyB0aGUgb25lIGhpdHRpbmcgdGhlIEFVVEhfRVJSIHRoZW4NCj4+Pj4+Pj4+ Pj4+IHRoZSBuZXcgY29udGV4dCBpcyBlc3RhYmxpc2hlZCBhbmQgdGhlbiBSRU5FVyB1c2luZyBp bnRlZ3JpdHkgc2VydmljZQ0KPj4+Pj4+Pj4+Pj4gaXMgcmV0cmllZCB3aXRoIHRoZSBuZXcgY29u dGV4dCB3aGljaCBnZXRzIEVSUl9TVEFMRV9DTElFTlRJRCB3aGljaA0KPj4+Pj4+Pj4+Pj4gdGhl biBjbGllbnQgcmVjb3ZlcnMgZnJvbS4gSWYgaXQncyBhbiBvcGVyYXRpb24gKEkgaGF2ZSBhIEdF VEFUVFIpDQo+Pj4+Pj4+Pj4+PiB0aGF0IGdldHMgQVVUSF9FUlIsIHRoZW4gaXQgZ2V0cyBuZXcg Y29udGV4dCBhbmQgaXMgcmV0cmllZCB1c2luZyBub25lDQo+Pj4+Pj4+Pj4+PiBzZXJ2aWNlLiBU aGVuIFJFTkVXIGdldHMgaXRzIG93biBBVVRIX0VSUiBhcyBpdCB1c2VzIGEgZGlmZmVyZW50DQo+ Pj4+Pj4+Pj4+PiBjb250ZXh0LCBhIG5ldyBjb250ZXh0IGlzIGdvdHRlbiwgUkVORVcgaXMgcmV0 cmllZCBvdmVyIGludGVncml0eSBhbmQNCj4+Pj4+Pj4+Pj4+IGdldHMgRVJSX1NUQUxFX0NMSUVO VElEIHdoaWNoIGl0IHJlY292ZXJzIGZyb20uDQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBJZiBv bmUgb3BlcmF0aW9uIGlzIGFsbG93ZWQgdG8gY29tcGxldGUsIHRoZW4NCj4+Pj4+Pj4+Pj4gdGhl IG90aGVyIHdpbGwgYWx3YXlzIHJlY29nbml6ZSB0aGF0IGFub3RoZXINCj4+Pj4+Pj4+Pj4gZnJl c2ggR1NTIGNvbnRleHQgaXMgbmVlZGVkLiBJZiB0d28gYXJlIHNlbnQNCj4+Pj4+Pj4+Pj4gYXQg dGhlIHNhbWUgdGltZSwgdGhleSByYWNlIGFuZCBvbmUgYWx3YXlzDQo+Pj4+Pj4+Pj4+IGZhaWxz Lg0KPj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4gSGVsZW4ncyB0ZXN0IGluY2x1ZGVzIGEgc2Vjb25k IGlkbGUgbW91bnQgcG9pbnQNCj4+Pj4+Pj4+Pj4gKHNlYz1rcmI1aSkgYW5kIG1heWJlIHRoYXQg aXMgbmVlZGVkIHRvIHRyaWdnZXINCj4+Pj4+Pj4+Pj4gdGhlIHJhY2U/DQo+Pj4+Pj4+Pj4gDQo+ Pj4+Pj4+Pj4gQ2h1Y2ssIGFueSBjaGFuY2UgdG8gZ2V0ICJycGNkZWJ1ZyAtbSBycGMgYXV0aCIg b3V0cHV0IGR1cmluZyB0aGUNCj4+Pj4+Pj4+PiBmYWlsdXJlIChnc3NkIG9wdGlvbmFsbHkpIChp IHJlYWxpemUgdGhhdCBpdCBtaWdodCBhbHRlciB0aGUgdGltaW5ncw0KPj4+Pj4+Pj4+IGFuZCBu b3QgaGl0IHRoZSBpc3N1ZSBidXQgd29ydGggYSBzaG90KT8NCj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4g SSdtIHN1cmUgdGhhdCdzIGZpbmUuIEFuIGludGVybmFsIHRlc3RlciBoaXQgdGhpcywNCj4+Pj4+ Pj4+IG5vdCBhIGN1c3RvbWVyLCBzbyBJIHdpbGwgYXNrLg0KPj4+Pj4+Pj4gDQo+Pj4+Pj4+PiBJ IGFncmVlLCB0aG91Z2gsIHRoYXQgdGltaW5nIG1pZ2h0IGJlIGEgcHJvYmxlbToNCj4+Pj4+Pj4+ IHRoZXNlIHN5c3RlbXMgYWxsIGhhdmUgcmVhbCBzZXJpYWwgY29uc29sZXMgdmlhDQo+Pj4+Pj4+ PiBpTE9NLCBzbyAvdi9sL20gdHJhZmZpYyBkb2VzIGJyaW5nIGV2ZXJ5dGhpbmcgdG8NCj4+Pj4+ Pj4+IGEgc3RhbmRzdGlsbC4NCj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4gTWVhbndoaWxlLCB3aGF0J3Mg eW91J3JlIG9waW5pb24gYWJvdXQgQVVUSF9GQUlMRUQ/DQo+Pj4+Pj4+PiBTaG91bGQgdGhlIHNl cnZlciByZXR1cm4gUlBDU0VDX0dTU19DVFhQUk9CTEVNDQo+Pj4+Pj4+PiBpbiB0aGlzIGNhc2Ug aW5zdGVhZD8gSWYgaXQgZGlkLCBkbyB5b3UgdGhpbmsNCj4+Pj4+Pj4+IHRoZSBMaW51eCBjbGll bnQgd291bGQgcmVjb3ZlciBieSBjcmVhdGluZyBhDQo+Pj4+Pj4+PiByZXBsYWNlbWVudCBHU1Mg Y29udGV4dD8NCj4+Pj4+Pj4gDQo+Pj4+Pj4+IEFoLCB5ZXMsIEkgZXF1YXRlZCBBVVRIX0ZBSUxF RCBBbmQgQVVUSF9FUlJPUiBpbiBteSBtaW5kLiBJZiBjbGllbnQNCj4+Pj4+Pj4gcmVjZWl2ZXMg dGhlIHJlYXNvbiBhcyBBVVRIX0ZBSUxFRCBhcyBvcHBvc2UgdG8gQ1RYUFJPQkxFTSBpdCB3aWxs DQo+Pj4+Pj4+IGZhaWwgd2l0aCBFSU8gZXJyb3IgYW5kIHdpbGwgbm90IHRyeSB0byBjcmVhdGUg YSBuZXcgR1NTIGNvbnRleHQuIFNvDQo+Pj4+Pj4+IHllcywgSSBiZWxpZXZlIGl0IHdvdWxkIGhl bHAgaWYgdGhlIHNlcnZlciByZXR1cm5zIGFueSBvZiB0aGUNCj4+Pj4+Pj4gZm9sbG93aW5nIGVy cm9yczoNCj4+Pj4+Pj4gICAgICAgICAgICAgY2FzZSBSUENfQVVUSF9SRUpFQ1RFRENSRUQ6DQo+ Pj4+Pj4+ICAgICAgICAgICAgIGNhc2UgUlBDX0FVVEhfUkVKRUNURURWRVJGOg0KPj4+Pj4+PiAg ICAgICAgICAgICBjYXNlIFJQQ1NFQ19HU1NfQ1JFRFBST0JMRU06DQo+Pj4+Pj4+ICAgICAgICAg ICAgIGNhc2UgUlBDU0VDX0dTU19DVFhQUk9CTEVNOg0KPj4+Pj4+PiANCj4+Pj4+Pj4gdGhlbiB0 aGUgY2xpZW50IHdpbGwgcmVjcmVhdGUgdGhlIGNvbnRleHQuDQo+Pj4+Pj4gDQo+Pj4+Pj4gQWxz byBpbiBteSB0ZXN0aW5nLCBJIGNhbiBzZWUgdGhhdCBjcmVkZW50aWFsIGNhY2hlIGlzIHBlciBn c3MgZmxhdm9yLg0KPj4+Pj4+IEp1c3QgdG8gY2hlY2ssIHdoYXQga2VybmVsIHZlcnNpb24gaXMg dGhpcyBwcm9ibGVtIGVuY291bnRlcmVkIG9uIChJDQo+Pj4+Pj4ga25vdyB5b3Ugc2FpZCB1cHN0 cmVhbSkgYnV0IEkganVzdCB3YW50IHRvIGRvdWJsZSBjaGVjayBzbyB0aGF0IEkgY2FuDQo+Pj4+ Pj4gbG9vayBhdCB0aGUgY29ycmVjdCBzb3VyY2UgY29kZS4NCj4+Pj4+IA0KPj4+Pj4gdjQuMS4x MiAoc3RhYmxlKSBJIHRoaW5rLg0KPj4+PiANCj4+Pj4gQWxzbywgY2FuIHlvdSBzaGFyZSB0aGUg bmV0d29yayB0cmFjZT8NCj4+PiANCj4+PiBIaSBDaHVjaywNCj4+PiANCj4+PiBJIHdhcyBmaW5h bGx5IGFibGUgdG8gcmVwcm9kdWNlIHRoZSBjb25kaXRpb24geW91IHdlcmUgc2VlaW5nIChpLmUu LA0KPj4+IHRoZSB1c2Ugb2YgdGhlIHNhbWUgY29udGV4dCBmb3IgZGlmZmVyZW50IGdzcyBzZXJ2 aWNlcykuDQo+Pj4gDQo+Pj4gSSBlbmFibGVkIHJwY2RlYnVnIHJwYyBhdXRoIGFuZCBJIGNhbiBz ZWUgdGhhdCB0aGUgMm5kIHJlcXVlc3QgZW5kcyB1cA0KPj4+IGZpbmRpbmcgYSBnc3NfdXBjYWxs IG1lc3NhZ2UgYmVjYXVzZSBpdCdzIGp1c3QgbWF0Y2hlZCBieSB0aGUgdWlkLg0KPj4+IFRoZXJl IGlzIGV2ZW4gYSBjb21tZW50IGluIGF1dGhfZ3NzL2F1dGhfZ3NzLmMgaW4gZ3NzX2FkZF9tc2co KSBzYXlpbmcNCj4+PiB0aGF0IGlmIHRoZXJlIGlzIHVwY2FsbCBmb3IgYW4gdWlkIHRoZW4gaXQg d29uJ3QgYWRkIGFub3RoZXIgdXBjYWxsLg0KPj4+IFNvIEkgdGhpbmsgdGhlIGRlY2lzaW9uIGlz IG1hZGUgcmlnaHQgdGhlcmUgdG8gc2hhcmUgdGhlIHNhbWUgY29udGV4dA0KPj4+IG5vIG1hdHRl ciB0aGUgZ3NzIHNlcnZpY2UuDQo+PiANCj4+IElmIEkgdW5kZXJzdGFuZCBjb3JyZWN0bHksIHRo YXQncyBqdXN0IHdoYXQgQW5keSBwcmVkaWN0ZWQuDQo+PiANCj4+IFRoYXQgY2hlY2sgbmVlZHMg dG8gYmUgY2hhbmdlZCB0byBhbGxvdyBhbm90aGVyIHVwY2FsbCB0byBiZQ0KPj4gcXVldWVkIGlm IHRoZSBVSUQgbWF0Y2hlcyBidXQgdGhlIEdTUyBzZXJ2aWNlIGRvZXMgbm90Lg0KPiANCj4gWWVz LCB3ZSBuZWVkIHRvIGVuc3VyZSB0aGF0IHRoZSBzZXJ2aWNlIGlzIHNldCBldmVuIHRob3VnaCBp dCBpcyBpZ25vcmVkIGJ5IHRoZSBzZXJ2ZXIuDQoNCuKApnNlcnZpY2UgaXMgc2V0IGluIHRoZSBn c3NfdXBjYWxsX21zZ+KApg0KDQo+IA0KPiDigJQ+QW5keQ0KPiANCj4+IA0KPj4gDQo+PiAtLQ0K Pj4gQ2h1Y2sgTGV2ZXINCg0K ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-29 17:32 ` Adamson, Andy @ 2016-07-29 22:24 ` Olga Kornievskaia 0 siblings, 0 replies; 25+ messages in thread From: Olga Kornievskaia @ 2016-07-29 22:24 UTC (permalink / raw) To: Adamson, Andy; +Cc: Chuck Lever, Linux NFS Mailing List On Fri, Jul 29, 2016 at 1:32 PM, Adamson, Andy <William.Adamson@netapp.com> wrote: > >> On Jul 29, 2016, at 1:07 PM, Adamson, Andy <William.Adamson@netapp.com> wrote: >> >>> >>> On Jul 29, 2016, at 12:38 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>> >>>> >>>> On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> >>>> On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>> >>>>>> >>>>>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>> >>>>>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>> >>>>>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>>> >>>>>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>>>>>>>> <William.Adamson@netapp.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Andy- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>>>>>>>> report. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>>>>>>>> and NFSv4.1 mount points. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>>>>>>>> the network capture. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>>>>>>>> >>>>>>>>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>>>>>>>> lease management operations, which is bound to the client's >>>>>>>>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>>>>>>>> Call this GSS context 1. >>>>>>>>>>>>>> >>>>>>>>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>>>>>>>> established for NFS operations associated with that user. >>>>>>>>>>>>>> Since the reproducer is running as root, this context is >>>>>>>>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>>>>>>>> >>>>>>>>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>>>>>>>> connection with the server, and performs a RENEW >>>>>>>>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>>>>>>>> problem at all). >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>>>>>>>> service "none" is used for this operation, since it is on >>>>>>>>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The RPC is accepted, but the server reports >>>>>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client responds by attempting state recovery. The >>>>>>>>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>>>>>>>> credential cache again and finds the recently established >>>>>>>>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>>>>>>>> 3 with GSS service "integrity." >>>>>>>>>>>>>> >>>>>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>>>>>>>> the client reports that "check lease failed" and >>>>>>>>>>>>>> terminates state recovery. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>>>>>>>> stateid with predictable results. The client again tries >>>>>>>>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>>>>>>>> management operations, attempt to use the same GSS context >>>>>>>>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>>>>>>>> services. >>>>>>>>>>>>> >>>>>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> The key issue seems to be why, when the mount >>>>>>>>>>>>>> is first established, the client is correctly able to >>>>>>>>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>>>>>>>> context with two different GSS services. >>>>>>>>>>>>> >>>>>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>>>>>>>> >>>>>>>>>>>> I agree with Andy. It must be a tight race. >>>>>>>>>>> >>>>>>>>>>> In one capture I see something like this after >>>>>>>>>>> the server restarts: >>>>>>>>>>> >>>>>>>>>>> SYN >>>>>>>>>>> SYN, ACK >>>>>>>>>>> ACK >>>>>>>>>>> C WRITE >>>>>>>>>>> C SEQUENCE >>>>>>>>>>> R WRITE -> CTX_PROBLEM >>>>>>>>>>> R SEQUENCE -> CTX_PROBLEM >>>>>>>>>>> C NULL (KRB5_AP_REQ) >>>>>>>>>>> R NULL (KRB5_AP_REP) >>>>>>>>>>> C WRITE >>>>>>>>>>> C SEQUENCE >>>>>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>>>>>>>> R SEQUENCE -> AUTH_FAILED >>>>>>>>>>> >>>>>>>>>>> Andy's theory neatly explains this behavior. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> I have tried to reproduce >>>>>>>>>>>> your scenario and in my tests of rebooting the server all recover >>>>>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>>>>>>>> the new context is established and then RENEW using integrity service >>>>>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>>>>>>>> >>>>>>>>>>> If one operation is allowed to complete, then >>>>>>>>>>> the other will always recognize that another >>>>>>>>>>> fresh GSS context is needed. If two are sent >>>>>>>>>>> at the same time, they race and one always >>>>>>>>>>> fails. >>>>>>>>>>> >>>>>>>>>>> Helen's test includes a second idle mount point >>>>>>>>>>> (sec=krb5i) and maybe that is needed to trigger >>>>>>>>>>> the race? >>>>>>>>>> >>>>>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>>>>>>>> failure (gssd optionally) (i realize that it might alter the timings >>>>>>>>>> and not hit the issue but worth a shot)? >>>>>>>>> >>>>>>>>> I'm sure that's fine. An internal tester hit this, >>>>>>>>> not a customer, so I will ask. >>>>>>>>> >>>>>>>>> I agree, though, that timing might be a problem: >>>>>>>>> these systems all have real serial consoles via >>>>>>>>> iLOM, so /v/l/m traffic does bring everything to >>>>>>>>> a standstill. >>>>>>>>> >>>>>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? >>>>>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM >>>>>>>>> in this case instead? If it did, do you think >>>>>>>>> the Linux client would recover by creating a >>>>>>>>> replacement GSS context? >>>>>>>> >>>>>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >>>>>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >>>>>>>> fail with EIO error and will not try to create a new GSS context. So >>>>>>>> yes, I believe it would help if the server returns any of the >>>>>>>> following errors: >>>>>>>> case RPC_AUTH_REJECTEDCRED: >>>>>>>> case RPC_AUTH_REJECTEDVERF: >>>>>>>> case RPCSEC_GSS_CREDPROBLEM: >>>>>>>> case RPCSEC_GSS_CTXPROBLEM: >>>>>>>> >>>>>>>> then the client will recreate the context. >>>>>>> >>>>>>> Also in my testing, I can see that credential cache is per gss flavor. >>>>>>> Just to check, what kernel version is this problem encountered on (I >>>>>>> know you said upstream) but I just want to double check so that I can >>>>>>> look at the correct source code. >>>>>> >>>>>> v4.1.12 (stable) I think. >>>>> >>>>> Also, can you share the network trace? >>>> >>>> Hi Chuck, >>>> >>>> I was finally able to reproduce the condition you were seeing (i.e., >>>> the use of the same context for different gss services). >>>> >>>> I enabled rpcdebug rpc auth and I can see that the 2nd request ends up >>>> finding a gss_upcall message because it's just matched by the uid. >>>> There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying >>>> that if there is upcall for an uid then it won't add another upcall. >>>> So I think the decision is made right there to share the same context >>>> no matter the gss service. >>> >>> If I understand correctly, that's just what Andy predicted. >>> >>> That check needs to be changed to allow another upcall to be >>> queued if the UID matches but the GSS service does not. >> >> Yes, we need to ensure that the service is set even though it is ignored by the server. > > …service is set in the gss_upcall_msg… > I believe a change to accommodate this would have compatibility issues. Currently, when the gssd creates a context and sends the information back to the kernel, the first thing in the buffer is the uid. The kernel parses the uid and then looks up the appropriate gss_upcall message. Right now no service information is there to do the matching. If we were to add gssd to sends up the service for which the context was established, then the old kernel won't work with the new nfs-utils. And new kernel won't work with old nfs-utils. >> >> —>Andy >> >>> >>> >>> -- >>> Chuck Lever > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-07-29 16:38 ` Chuck Lever 2016-07-29 17:07 ` Adamson, Andy @ 2016-08-02 18:06 ` J. Bruce Fields 2016-08-03 18:53 ` Adamson, Andy 2016-08-03 19:14 ` Chuck Lever 1 sibling, 2 replies; 25+ messages in thread From: J. Bruce Fields @ 2016-08-02 18:06 UTC (permalink / raw) To: Chuck Lever; +Cc: Olga Kornievskaia, Adamson, Andy, Linux NFS Mailing List On Fri, Jul 29, 2016 at 12:38:34PM -0400, Chuck Lever wrote: > > > On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > > > > On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>> > >>> > >>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>> > >>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>> > >>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>>> > >>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>>>> > >>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>>>>> > >>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy > >>>>>>>>> <William.Adamson@netapp.com> wrote: > >>>>>>>>>> > >>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Andy- > >>>>>>>>>>> > >>>>>>>>>>> Thanks for taking the time to discuss this with me. I've > >>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug > >>>>>>>>>>> report. > >>>>>>>>>>> > >>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts > >>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 > >>>>>>>>>>> and NFSv4.1 mount points. > >>>>>>>>>>> > >>>>>>>>>>> The reproducer is a simple program that generates one NFS > >>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots > >>>>>>>>>>> (or one cluster head fails over to the other and back). The > >>>>>>>>>>> goal of the reproducer is to identify problems with state > >>>>>>>>>>> recovery without a lot of other I/O going on to clutter up > >>>>>>>>>>> the network capture. > >>>>>>>>>>> > >>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount > >>>>>>>>>>> point, and the reproducer is run as root. We've found this > >>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. > >>>>>>>>>>> > >>>>>>>>>>> At mount time, the client establishes a GSS context for > >>>>>>>>>>> lease management operations, which is bound to the client's > >>>>>>>>>>> NFS service principal and uses GSS service "integrity." > >>>>>>>>>>> Call this GSS context 1. > >>>>>>>>>>> > >>>>>>>>>>> When the reproducer starts, a second GSS context is > >>>>>>>>>>> established for NFS operations associated with that user. > >>>>>>>>>>> Since the reproducer is running as root, this context is > >>>>>>>>>>> also bound to the client's NFS service principal, but it > >>>>>>>>>>> uses the GSS service "none" (reflecting the explicit > >>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. > >>>>>>>>>>> > >>>>>>>>>>> After the server reboots, the client re-establishes a TCP > >>>>>>>>>>> connection with the server, and performs a RENEW > >>>>>>>>>>> operation using context 1. Thanks to the server reboot, > >>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects > >>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. > >>>>>>>>>>> > >>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 > >>>>>>>>>>> NULL operation. Call this GSS context 3. > >>>>>>>>>>> > >>>>>>>>>>> Interestingly, the client does not resend the RENEW > >>>>>>>>>>> operation at this point (if it did, we wouldn't see this > >>>>>>>>>>> problem at all). > >>>>>>>>>>> > >>>>>>>>>>> The client then attempts to resume the reproducer workload. > >>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available > >>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, > >>>>>>>>>>> already bound to the client's NFS service principal. But GSS > >>>>>>>>>>> service "none" is used for this operation, since it is on > >>>>>>>>>>> behalf of the mount where sec=krb5 was specified. > >>>>>>>>>>> > >>>>>>>>>>> The RPC is accepted, but the server reports > >>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. > >>>>>>>>>>> > >>>>>>>>>>> The client responds by attempting state recovery. The > >>>>>>>>>>> first operation it tries is another RENEW. Since this is > >>>>>>>>>>> a lease management operation, the client looks in UID 0's > >>>>>>>>>>> credential cache again and finds the recently established > >>>>>>>>>>> context 3. It tries the RENEW operation using GSS context > >>>>>>>>>>> 3 with GSS service "integrity." > >>>>>>>>>>> > >>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and > >>>>>>>>>>> the client reports that "check lease failed" and > >>>>>>>>>>> terminates state recovery. > >>>>>>>>>>> > >>>>>>>>>>> The client re-drives the WRITE operation with the stale > >>>>>>>>>>> stateid with predictable results. The client again tries > >>>>>>>>>>> to recover state by sending a RENEW, and still uses the > >>>>>>>>>>> same GSS context 3 with service "integrity" and gets the > >>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop > >>>>>>>>>>> ensues, and the client mount point is deadlocked. > >>>>>>>>>>> > >>>>>>>>>>> Your analysis was that because the reproducer is run as > >>>>>>>>>>> root, both the reproducer's I/O operations, and lease > >>>>>>>>>>> management operations, attempt to use the same GSS context > >>>>>>>>>>> in UID 0's credential cache, but each uses different GSS > >>>>>>>>>>> services. > >>>>>>>>>> > >>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” > >>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. > >>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> The key issue seems to be why, when the mount > >>>>>>>>>>> is first established, the client is correctly able to > >>>>>>>>>>> establish two separate GSS contexts for UID 0; but after > >>>>>>>>>>> a server reboot, the client attempts to use the same GSS > >>>>>>>>>>> context with two different GSS services. > >>>>>>>>>> > >>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. > >>>>>>>>> > >>>>>>>>> I agree with Andy. It must be a tight race. > >>>>>>>> > >>>>>>>> In one capture I see something like this after > >>>>>>>> the server restarts: > >>>>>>>> > >>>>>>>> SYN > >>>>>>>> SYN, ACK > >>>>>>>> ACK > >>>>>>>> C WRITE > >>>>>>>> C SEQUENCE > >>>>>>>> R WRITE -> CTX_PROBLEM > >>>>>>>> R SEQUENCE -> CTX_PROBLEM > >>>>>>>> C NULL (KRB5_AP_REQ) > >>>>>>>> R NULL (KRB5_AP_REP) > >>>>>>>> C WRITE > >>>>>>>> C SEQUENCE > >>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID > >>>>>>>> R SEQUENCE -> AUTH_FAILED > >>>>>>>> > >>>>>>>> Andy's theory neatly explains this behavior. > >>>>>>>> > >>>>>>>> > >>>>>>>>> I have tried to reproduce > >>>>>>>>> your scenario and in my tests of rebooting the server all recover > >>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then > >>>>>>>>> the new context is established and then RENEW using integrity service > >>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which > >>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) > >>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none > >>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different > >>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and > >>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. > >>>>>>>> > >>>>>>>> If one operation is allowed to complete, then > >>>>>>>> the other will always recognize that another > >>>>>>>> fresh GSS context is needed. If two are sent > >>>>>>>> at the same time, they race and one always > >>>>>>>> fails. > >>>>>>>> > >>>>>>>> Helen's test includes a second idle mount point > >>>>>>>> (sec=krb5i) and maybe that is needed to trigger > >>>>>>>> the race? > >>>>>>> > >>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the > >>>>>>> failure (gssd optionally) (i realize that it might alter the timings > >>>>>>> and not hit the issue but worth a shot)? > >>>>>> > >>>>>> I'm sure that's fine. An internal tester hit this, > >>>>>> not a customer, so I will ask. > >>>>>> > >>>>>> I agree, though, that timing might be a problem: > >>>>>> these systems all have real serial consoles via > >>>>>> iLOM, so /v/l/m traffic does bring everything to > >>>>>> a standstill. > >>>>>> > >>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? > >>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM > >>>>>> in this case instead? If it did, do you think > >>>>>> the Linux client would recover by creating a > >>>>>> replacement GSS context? > >>>>> > >>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client > >>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will > >>>>> fail with EIO error and will not try to create a new GSS context. So > >>>>> yes, I believe it would help if the server returns any of the > >>>>> following errors: > >>>>> case RPC_AUTH_REJECTEDCRED: > >>>>> case RPC_AUTH_REJECTEDVERF: > >>>>> case RPCSEC_GSS_CREDPROBLEM: > >>>>> case RPCSEC_GSS_CTXPROBLEM: > >>>>> > >>>>> then the client will recreate the context. > >>>> > >>>> Also in my testing, I can see that credential cache is per gss flavor. > >>>> Just to check, what kernel version is this problem encountered on (I > >>>> know you said upstream) but I just want to double check so that I can > >>>> look at the correct source code. > >>> > >>> v4.1.12 (stable) I think. > >> > >> Also, can you share the network trace? > > > > Hi Chuck, > > > > I was finally able to reproduce the condition you were seeing (i.e., > > the use of the same context for different gss services). > > > > I enabled rpcdebug rpc auth and I can see that the 2nd request ends up > > finding a gss_upcall message because it's just matched by the uid. > > There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying > > that if there is upcall for an uid then it won't add another upcall. > > So I think the decision is made right there to share the same context > > no matter the gss service. > > If I understand correctly, that's just what Andy predicted. > > That check needs to be changed to allow another upcall to be > queued if the UID matches but the GSS service does not. You should be able to use the same context with different services. Apologies, I haven't caught up with the whole discussion above, this one point just jumped out at me. If you're trying to request a whole new gss context just so you can use, e.g., integrity instead of privacy, then something's wrong. --b. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-02 18:06 ` J. Bruce Fields @ 2016-08-03 18:53 ` Adamson, Andy 2016-08-03 19:56 ` Olga Kornievskaia 2016-08-03 19:14 ` Chuck Lever 1 sibling, 1 reply; 25+ messages in thread From: Adamson, Andy @ 2016-08-03 18:53 UTC (permalink / raw) To: J. Bruce Fields Cc: Chuck Lever, Olga Kornievskaia, Adamson, Andy, Linux NFS Mailing List DQo+IE9uIEF1ZyAyLCAyMDE2LCBhdCAyOjA2IFBNLCBKLiBCcnVjZSBGaWVsZHMgPGJmaWVsZHNA ZmllbGRzZXMub3JnPiB3cm90ZToNCj4gDQo+IE9uIEZyaSwgSnVsIDI5LCAyMDE2IGF0IDEyOjM4 OjM0UE0gLTA0MDAsIENodWNrIExldmVyIHdyb3RlOg0KPj4gDQo+Pj4gT24gSnVsIDI5LCAyMDE2 LCBhdCAxMjoyNyBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3cm90ZToN Cj4+PiANCj4+PiBPbiBNb24sIEp1bCAyNSwgMjAxNiBhdCAyOjE4IFBNLCBPbGdhIEtvcm5pZXZz a2FpYSA8YWdsb0B1bWljaC5lZHU+IHdyb3RlOg0KPj4+PiBPbiBUaHUsIEp1bCAyMSwgMjAxNiBh dCA1OjMyIFBNLCBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+ Pj4+PiANCj4+Pj4+IA0KPj4+Pj4+IE9uIEp1bCAyMSwgMjAxNiwgYXQgMTA6NDYgUE0sIE9sZ2Eg S29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4gDQo+Pj4+Pj4+IE9u IFRodSwgSnVsIDIxLCAyMDE2IGF0IDM6NTQgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVt aWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+PiBPbiBUaHUsIEp1bCAyMSwgMjAxNiBhdCAxOjU2IFBN LCBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+PiBPbiBKdWwgMjEsIDIwMTYsIGF0IDY6MDQgUE0sIE9sZ2EgS29ybmlldnNrYWlh IDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4gT24gVGh1LCBK dWwgMjEsIDIwMTYgYXQgMjo1NSBBTSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5j b20+IHdyb3RlOg0KPj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+IE9uIEp1bCAyMCwgMjAxNiwgYXQg Njo1NiBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3cm90ZToNCj4+Pj4+ Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4gT24gV2VkLCBKdWwgMjAsIDIwMTYgYXQgNToxNCBBTSwgQWRh bXNvbiwgQW5keQ0KPj4+Pj4+Pj4+Pj4gPFdpbGxpYW0uQWRhbXNvbkBuZXRhcHAuY29tPiB3cm90 ZToNCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gT24gSnVsIDE5LCAyMDE2LCBhdCAxMDo1 MSBBTSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+ Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gSGkgQW5keS0NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+ Pj4+Pj4+IFRoYW5rcyBmb3IgdGFraW5nIHRoZSB0aW1lIHRvIGRpc2N1c3MgdGhpcyB3aXRoIG1l LiBJJ3ZlDQo+Pj4+Pj4+Pj4+Pj4+IGNvcGllZCBsaW51eC1uZnMgdG8gbWFrZSB0aGlzIGUtbWFp bCBhbHNvIGFuIHVwc3RyZWFtIGJ1Zw0KPj4+Pj4+Pj4+Pj4+PiByZXBvcnQuDQo+Pj4+Pj4+Pj4+ Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBBcyB3ZSBzYXcgaW4gdGhlIG5ldHdvcmsgY2FwdHVyZSwgcmVj b3Zlcnkgb2YgR1NTIGNvbnRleHRzDQo+Pj4+Pj4+Pj4+Pj4+IGFmdGVyIGEgc2VydmVyIHJlYm9v dCBmYWlscyBpbiBjZXJ0YWluIGNhc2VzIHdpdGggTkZTdjQuMA0KPj4+Pj4+Pj4+Pj4+PiBhbmQg TkZTdjQuMSBtb3VudCBwb2ludHMuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBUaGUg cmVwcm9kdWNlciBpcyBhIHNpbXBsZSBwcm9ncmFtIHRoYXQgZ2VuZXJhdGVzIG9uZSBORlMNCj4+ Pj4+Pj4+Pj4+Pj4gV1JJVEUgcGVyaW9kaWNhbGx5LCBydW4gd2hpbGUgdGhlIHNlcnZlciByZXBl YXRlZGx5IHJlYm9vdHMNCj4+Pj4+Pj4+Pj4+Pj4gKG9yIG9uZSBjbHVzdGVyIGhlYWQgZmFpbHMg b3ZlciB0byB0aGUgb3RoZXIgYW5kIGJhY2spLiBUaGUNCj4+Pj4+Pj4+Pj4+Pj4gZ29hbCBvZiB0 aGUgcmVwcm9kdWNlciBpcyB0byBpZGVudGlmeSBwcm9ibGVtcyB3aXRoIHN0YXRlDQo+Pj4+Pj4+ Pj4+Pj4+IHJlY292ZXJ5IHdpdGhvdXQgYSBsb3Qgb2Ygb3RoZXIgSS9PIGdvaW5nIG9uIHRvIGNs dXR0ZXIgdXANCj4+Pj4+Pj4+Pj4+Pj4gdGhlIG5ldHdvcmsgY2FwdHVyZS4NCj4+Pj4+Pj4+Pj4+ Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IEluIHRoZSBmYWlsaW5nIGNhc2UsIHNlYz1rcmI1IGlzIHNwZWNp ZmllZCBvbiB0aGUgbW91bnQNCj4+Pj4+Pj4+Pj4+Pj4gcG9pbnQsIGFuZCB0aGUgcmVwcm9kdWNl ciBpcyBydW4gYXMgcm9vdC4gV2UndmUgZm91bmQgdGhpcw0KPj4+Pj4+Pj4+Pj4+PiBjb21iaW5h dGlvbiBmYWlscyB3aXRoIGJvdGggTkZTdjQuMCBhbmQgTkZTdjQuMS4NCj4+Pj4+Pj4+Pj4+Pj4g DQo+Pj4+Pj4+Pj4+Pj4+IEF0IG1vdW50IHRpbWUsIHRoZSBjbGllbnQgZXN0YWJsaXNoZXMgYSBH U1MgY29udGV4dCBmb3INCj4+Pj4+Pj4+Pj4+Pj4gbGVhc2UgbWFuYWdlbWVudCBvcGVyYXRpb25z LCB3aGljaCBpcyBib3VuZCB0byB0aGUgY2xpZW50J3MNCj4+Pj4+Pj4+Pj4+Pj4gTkZTIHNlcnZp Y2UgcHJpbmNpcGFsIGFuZCB1c2VzIEdTUyBzZXJ2aWNlICJpbnRlZ3JpdHkuIg0KPj4+Pj4+Pj4+ Pj4+PiBDYWxsIHRoaXMgR1NTIGNvbnRleHQgMS4NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+ Pj4+IFdoZW4gdGhlIHJlcHJvZHVjZXIgc3RhcnRzLCBhIHNlY29uZCBHU1MgY29udGV4dCBpcw0K Pj4+Pj4+Pj4+Pj4+PiBlc3RhYmxpc2hlZCBmb3IgTkZTIG9wZXJhdGlvbnMgYXNzb2NpYXRlZCB3 aXRoIHRoYXQgdXNlci4NCj4+Pj4+Pj4+Pj4+Pj4gU2luY2UgdGhlIHJlcHJvZHVjZXIgaXMgcnVu bmluZyBhcyByb290LCB0aGlzIGNvbnRleHQgaXMNCj4+Pj4+Pj4+Pj4+Pj4gYWxzbyBib3VuZCB0 byB0aGUgY2xpZW50J3MgTkZTIHNlcnZpY2UgcHJpbmNpcGFsLCBidXQgaXQNCj4+Pj4+Pj4+Pj4+ Pj4gdXNlcyB0aGUgR1NTIHNlcnZpY2UgIm5vbmUiIChyZWZsZWN0aW5nIHRoZSBleHBsaWNpdA0K Pj4+Pj4+Pj4+Pj4+PiByZXF1ZXN0IGZvciAic2VjPWtyYjUiKS4gQ2FsbCB0aGlzIEdTUyBjb250 ZXh0IDIuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBBZnRlciB0aGUgc2VydmVyIHJl Ym9vdHMsIHRoZSBjbGllbnQgcmUtZXN0YWJsaXNoZXMgYSBUQ1ANCj4+Pj4+Pj4+Pj4+Pj4gY29u bmVjdGlvbiB3aXRoIHRoZSBzZXJ2ZXIsIGFuZCBwZXJmb3JtcyBhIFJFTkVXDQo+Pj4+Pj4+Pj4+ Pj4+IG9wZXJhdGlvbiB1c2luZyBjb250ZXh0IDEuIFRoYW5rcyB0byB0aGUgc2VydmVyIHJlYm9v dCwNCj4+Pj4+Pj4+Pj4+Pj4gY29udGV4dHMgMSBhbmQgMiBhcmUgbm93IHN0YWxlLiBUaGUgc2Vy dmVyIHRodXMgcmVqZWN0cw0KPj4+Pj4+Pj4+Pj4+PiB0aGUgUlBDIHdpdGggUlBDU0VDX0dTU19D VFhQUk9CTEVNLg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCBwZXJm b3JtcyBhIEdTU19JTklUX1NFQ19DT05URVhUIHZpYSBhbiBORlN2NA0KPj4+Pj4+Pj4+Pj4+PiBO VUxMIG9wZXJhdGlvbi4gQ2FsbCB0aGlzIEdTUyBjb250ZXh0IDMuDQo+Pj4+Pj4+Pj4+Pj4+IA0K Pj4+Pj4+Pj4+Pj4+PiBJbnRlcmVzdGluZ2x5LCB0aGUgY2xpZW50IGRvZXMgbm90IHJlc2VuZCB0 aGUgUkVORVcNCj4+Pj4+Pj4+Pj4+Pj4gb3BlcmF0aW9uIGF0IHRoaXMgcG9pbnQgKGlmIGl0IGRp ZCwgd2Ugd291bGRuJ3Qgc2VlIHRoaXMNCj4+Pj4+Pj4+Pj4+Pj4gcHJvYmxlbSBhdCBhbGwpLg0K Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCB0aGVuIGF0dGVtcHRzIHRv IHJlc3VtZSB0aGUgcmVwcm9kdWNlciB3b3JrbG9hZC4NCj4+Pj4+Pj4+Pj4+Pj4gSXQgc2VuZHMg YW4gTkZTdjQgV1JJVEUgb3BlcmF0aW9uLCB1c2luZyB0aGUgZmlyc3QgYXZhaWxhYmxlDQo+Pj4+ Pj4+Pj4+Pj4+IEdTUyBjb250ZXh0IGluIFVJRCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgd2hpY2gg aXMgY29udGV4dCAzLA0KPj4+Pj4+Pj4+Pj4+PiBhbHJlYWR5IGJvdW5kIHRvIHRoZSBjbGllbnQn cyBORlMgc2VydmljZSBwcmluY2lwYWwuIEJ1dCBHU1MNCj4+Pj4+Pj4+Pj4+Pj4gc2VydmljZSAi bm9uZSIgaXMgdXNlZCBmb3IgdGhpcyBvcGVyYXRpb24sIHNpbmNlIGl0IGlzIG9uDQo+Pj4+Pj4+ Pj4+Pj4+IGJlaGFsZiBvZiB0aGUgbW91bnQgd2hlcmUgc2VjPWtyYjUgd2FzIHNwZWNpZmllZC4N Cj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IFRoZSBSUEMgaXMgYWNjZXB0ZWQsIGJ1dCB0 aGUgc2VydmVyIHJlcG9ydHMNCj4+Pj4+Pj4+Pj4+Pj4gTkZTNEVSUl9TVEFMRV9TVEFURUlELCBz aW5jZSBpdCBoYXMgcmVjZW50bHkgcmVib290ZWQuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+PiBUaGUgY2xpZW50IHJlc3BvbmRzIGJ5IGF0dGVtcHRpbmcgc3RhdGUgcmVjb3ZlcnkuIFRo ZQ0KPj4+Pj4+Pj4+Pj4+PiBmaXJzdCBvcGVyYXRpb24gaXQgdHJpZXMgaXMgYW5vdGhlciBSRU5F Vy4gU2luY2UgdGhpcyBpcw0KPj4+Pj4+Pj4+Pj4+PiBhIGxlYXNlIG1hbmFnZW1lbnQgb3BlcmF0 aW9uLCB0aGUgY2xpZW50IGxvb2tzIGluIFVJRCAwJ3MNCj4+Pj4+Pj4+Pj4+Pj4gY3JlZGVudGlh bCBjYWNoZSBhZ2FpbiBhbmQgZmluZHMgdGhlIHJlY2VudGx5IGVzdGFibGlzaGVkDQo+Pj4+Pj4+ Pj4+Pj4+IGNvbnRleHQgMy4gSXQgdHJpZXMgdGhlIFJFTkVXIG9wZXJhdGlvbiB1c2luZyBHU1Mg Y29udGV4dA0KPj4+Pj4+Pj4+Pj4+PiAzIHdpdGggR1NTIHNlcnZpY2UgImludGVncml0eS4iDQo+ Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBUaGUgc2VydmVyIHJlamVjdHMgdGhlIFJFTkVX IFJQQyB3aXRoIEFVVEhfRkFJTEVELCBhbmQNCj4+Pj4+Pj4+Pj4+Pj4gdGhlIGNsaWVudCByZXBv cnRzIHRoYXQgImNoZWNrIGxlYXNlIGZhaWxlZCIgYW5kDQo+Pj4+Pj4+Pj4+Pj4+IHRlcm1pbmF0 ZXMgc3RhdGUgcmVjb3ZlcnkuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBUaGUgY2xp ZW50IHJlLWRyaXZlcyB0aGUgV1JJVEUgb3BlcmF0aW9uIHdpdGggdGhlIHN0YWxlDQo+Pj4+Pj4+ Pj4+Pj4+IHN0YXRlaWQgd2l0aCBwcmVkaWN0YWJsZSByZXN1bHRzLiBUaGUgY2xpZW50IGFnYWlu IHRyaWVzDQo+Pj4+Pj4+Pj4+Pj4+IHRvIHJlY292ZXIgc3RhdGUgYnkgc2VuZGluZyBhIFJFTkVX LCBhbmQgc3RpbGwgdXNlcyB0aGUNCj4+Pj4+Pj4+Pj4+Pj4gc2FtZSBHU1MgY29udGV4dCAzIHdp dGggc2VydmljZSAiaW50ZWdyaXR5IiBhbmQgZ2V0cyB0aGUNCj4+Pj4+Pj4+Pj4+Pj4gc2FtZSBy ZXN1bHQuIEEgKHBlcmhhcHMgc2xvdy1tb3Rpb24pIFNUQUxFX1NUQVRFSUQgbG9vcA0KPj4+Pj4+ Pj4+Pj4+PiBlbnN1ZXMsIGFuZCB0aGUgY2xpZW50IG1vdW50IHBvaW50IGlzIGRlYWRsb2NrZWQu DQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+PiBZb3VyIGFuYWx5c2lzIHdhcyB0aGF0IGJl Y2F1c2UgdGhlIHJlcHJvZHVjZXIgaXMgcnVuIGFzDQo+Pj4+Pj4+Pj4+Pj4+IHJvb3QsIGJvdGgg dGhlIHJlcHJvZHVjZXIncyBJL08gb3BlcmF0aW9ucywgYW5kIGxlYXNlDQo+Pj4+Pj4+Pj4+Pj4+ IG1hbmFnZW1lbnQgb3BlcmF0aW9ucywgYXR0ZW1wdCB0byB1c2UgdGhlIHNhbWUgR1NTIGNvbnRl eHQNCj4+Pj4+Pj4+Pj4+Pj4gaW4gVUlEIDAncyBjcmVkZW50aWFsIGNhY2hlLCBidXQgZWFjaCB1 c2VzIGRpZmZlcmVudCBHU1MNCj4+Pj4+Pj4+Pj4+Pj4gc2VydmljZXMuDQo+Pj4+Pj4+Pj4+Pj4g DQo+Pj4+Pj4+Pj4+Pj4gQXMgUkZDMjIwMyBzdGF0ZXMsICJJbiBhIGNyZWF0aW9uIHJlcXVlc3Qs IHRoZSBzZXFfbnVtIGFuZCBzZXJ2aWNlIGZpZWxkcyBhcmUgdW5kZWZpbmVkIGFuZCBib3RoIG11 c3QgYmUgaWdub3JlZCBieSB0aGUgc2VydmVy4oCdDQo+Pj4+Pj4+Pj4+Pj4gU28gYSBjb250ZXh0 IGNyZWF0aW9uIHJlcXVlc3Qgd2hpbGUga2lja2VkIG9mZiBieSBhbiBvcGVyYXRpb24gd2l0aCBh IHNlcnZpY2UgYXR0YWNoZWQgKGUuZy4gV1JJVEUgdXNlcyBycGNfZ3NzX3N2Y19ub25lIGFuZCBS RU5FVyB1c2VzIHJwY19nc3Nfc3ZjX2ludGVncml0eSksIGNhbiBiZSB1c2VkIGJ5IGVpdGhlciBz ZXJ2aWNlIGxldmVsLg0KPj4+Pj4+Pj4+Pj4+IEFGQUlDUyBhIHNpbmdsZSBHU1MgY29udGV4dCBj b3VsZCBpbiB0aGVvcnkgYmUgdXNlZCBmb3IgYWxsIHNlcnZpY2UgbGV2ZWxzLCBidXQgaW4gcHJh Y3RpY2UsIEdTUyBjb250ZXh0cyBhcmUgcmVzdHJpY3RlZCB0byBhIHNlcnZpY2UgbGV2ZWwgKGJ5 IGNsaWVudD8gYnkgc2VydmVyPyApIG9uY2UgdGhleSBhcmUgdXNlZC4NCj4+Pj4+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gVGhlIGtleSBpc3N1ZSBzZWVtcyB0byBiZSB3 aHksIHdoZW4gdGhlIG1vdW50DQo+Pj4+Pj4+Pj4+Pj4+IGlzIGZpcnN0IGVzdGFibGlzaGVkLCB0 aGUgY2xpZW50IGlzIGNvcnJlY3RseSBhYmxlIHRvDQo+Pj4+Pj4+Pj4+Pj4+IGVzdGFibGlzaCB0 d28gc2VwYXJhdGUgR1NTIGNvbnRleHRzIGZvciBVSUQgMDsgYnV0IGFmdGVyDQo+Pj4+Pj4+Pj4+ Pj4+IGEgc2VydmVyIHJlYm9vdCwgdGhlIGNsaWVudCBhdHRlbXB0cyB0byB1c2UgdGhlIHNhbWUg R1NTDQo+Pj4+Pj4+Pj4+Pj4+IGNvbnRleHQgd2l0aCB0d28gZGlmZmVyZW50IEdTUyBzZXJ2aWNl cy4NCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBJIHNwZWN1bGF0ZSB0aGF0IGl0IGlzIGEg cmFjZSBiZXR3ZWVuIHRoZSBXUklURSBhbmQgdGhlIFJFTkVXIHRvIHVzZSB0aGUgc2FtZSBuZXds eSBjcmVhdGVkIEdTUyBjb250ZXh0IHRoYXQgaGFzIG5vdCBiZWVuIHVzZWQgeWV0LCBhbmQgc28g aGFzIG5vIGFzc2lnbmVkIHNlcnZpY2UgbGV2ZWwsIGFuZCB0aGUgdHdvIHJlcXVlc3RzIHJhY2Ug dG8gc2V0IHRoZSBzZXJ2aWNlIGxldmVsLg0KPj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+PiBJIGFn cmVlIHdpdGggQW5keS4gSXQgbXVzdCBiZSBhIHRpZ2h0IHJhY2UuDQo+Pj4+Pj4+Pj4+IA0KPj4+ Pj4+Pj4+PiBJbiBvbmUgY2FwdHVyZSBJIHNlZSBzb21ldGhpbmcgbGlrZSB0aGlzIGFmdGVyDQo+ Pj4+Pj4+Pj4+IHRoZSBzZXJ2ZXIgcmVzdGFydHM6DQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBT WU4NCj4+Pj4+Pj4+Pj4gU1lOLCBBQ0sNCj4+Pj4+Pj4+Pj4gQUNLDQo+Pj4+Pj4+Pj4+IEMgV1JJ VEUNCj4+Pj4+Pj4+Pj4gQyBTRVFVRU5DRQ0KPj4+Pj4+Pj4+PiBSIFdSSVRFIC0+IENUWF9QUk9C TEVNDQo+Pj4+Pj4+Pj4+IFIgU0VRVUVOQ0UgLT4gQ1RYX1BST0JMRU0NCj4+Pj4+Pj4+Pj4gQyBO VUxMIChLUkI1X0FQX1JFUSkNCj4+Pj4+Pj4+Pj4gUiBOVUxMIChLUkI1X0FQX1JFUCkNCj4+Pj4+ Pj4+Pj4gQyBXUklURQ0KPj4+Pj4+Pj4+PiBDIFNFUVVFTkNFDQo+Pj4+Pj4+Pj4+IFIgV1JJVEUg LT4gTkZTNEVSUl9TVEFMRV9TVEFURUlEDQo+Pj4+Pj4+Pj4+IFIgU0VRVUVOQ0UgLT4gQVVUSF9G QUlMRUQNCj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+IEFuZHkncyB0aGVvcnkgbmVhdGx5IGV4cGxh aW5zIHRoaXMgYmVoYXZpb3IuDQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ IEkgaGF2ZSB0cmllZCB0byByZXByb2R1Y2UNCj4+Pj4+Pj4+Pj4+IHlvdXIgc2NlbmFyaW8gYW5k IGluIG15IHRlc3RzIG9mIHJlYm9vdGluZyB0aGUgc2VydmVyIGFsbCByZWNvdmVyDQo+Pj4+Pj4+ Pj4+PiBjb3JyZWN0bHkuIEluIG15IGNhc2UsIGlmIFJFTkVXIHdhcyB0aGUgb25lIGhpdHRpbmcg dGhlIEFVVEhfRVJSIHRoZW4NCj4+Pj4+Pj4+Pj4+IHRoZSBuZXcgY29udGV4dCBpcyBlc3RhYmxp c2hlZCBhbmQgdGhlbiBSRU5FVyB1c2luZyBpbnRlZ3JpdHkgc2VydmljZQ0KPj4+Pj4+Pj4+Pj4g aXMgcmV0cmllZCB3aXRoIHRoZSBuZXcgY29udGV4dCB3aGljaCBnZXRzIEVSUl9TVEFMRV9DTElF TlRJRCB3aGljaA0KPj4+Pj4+Pj4+Pj4gdGhlbiBjbGllbnQgcmVjb3ZlcnMgZnJvbS4gSWYgaXQn cyBhbiBvcGVyYXRpb24gKEkgaGF2ZSBhIEdFVEFUVFIpDQo+Pj4+Pj4+Pj4+PiB0aGF0IGdldHMg QVVUSF9FUlIsIHRoZW4gaXQgZ2V0cyBuZXcgY29udGV4dCBhbmQgaXMgcmV0cmllZCB1c2luZyBu b25lDQo+Pj4+Pj4+Pj4+PiBzZXJ2aWNlLiBUaGVuIFJFTkVXIGdldHMgaXRzIG93biBBVVRIX0VS UiBhcyBpdCB1c2VzIGEgZGlmZmVyZW50DQo+Pj4+Pj4+Pj4+PiBjb250ZXh0LCBhIG5ldyBjb250 ZXh0IGlzIGdvdHRlbiwgUkVORVcgaXMgcmV0cmllZCBvdmVyIGludGVncml0eSBhbmQNCj4+Pj4+ Pj4+Pj4+IGdldHMgRVJSX1NUQUxFX0NMSUVOVElEIHdoaWNoIGl0IHJlY292ZXJzIGZyb20uDQo+ Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBJZiBvbmUgb3BlcmF0aW9uIGlzIGFsbG93ZWQgdG8gY29t cGxldGUsIHRoZW4NCj4+Pj4+Pj4+Pj4gdGhlIG90aGVyIHdpbGwgYWx3YXlzIHJlY29nbml6ZSB0 aGF0IGFub3RoZXINCj4+Pj4+Pj4+Pj4gZnJlc2ggR1NTIGNvbnRleHQgaXMgbmVlZGVkLiBJZiB0 d28gYXJlIHNlbnQNCj4+Pj4+Pj4+Pj4gYXQgdGhlIHNhbWUgdGltZSwgdGhleSByYWNlIGFuZCBv bmUgYWx3YXlzDQo+Pj4+Pj4+Pj4+IGZhaWxzLg0KPj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4gSGVs ZW4ncyB0ZXN0IGluY2x1ZGVzIGEgc2Vjb25kIGlkbGUgbW91bnQgcG9pbnQNCj4+Pj4+Pj4+Pj4g KHNlYz1rcmI1aSkgYW5kIG1heWJlIHRoYXQgaXMgbmVlZGVkIHRvIHRyaWdnZXINCj4+Pj4+Pj4+ Pj4gdGhlIHJhY2U/DQo+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4gQ2h1Y2ssIGFueSBjaGFuY2UgdG8g Z2V0ICJycGNkZWJ1ZyAtbSBycGMgYXV0aCIgb3V0cHV0IGR1cmluZyB0aGUNCj4+Pj4+Pj4+PiBm YWlsdXJlIChnc3NkIG9wdGlvbmFsbHkpIChpIHJlYWxpemUgdGhhdCBpdCBtaWdodCBhbHRlciB0 aGUgdGltaW5ncw0KPj4+Pj4+Pj4+IGFuZCBub3QgaGl0IHRoZSBpc3N1ZSBidXQgd29ydGggYSBz aG90KT8NCj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4gSSdtIHN1cmUgdGhhdCdzIGZpbmUuIEFuIGludGVy bmFsIHRlc3RlciBoaXQgdGhpcywNCj4+Pj4+Pj4+IG5vdCBhIGN1c3RvbWVyLCBzbyBJIHdpbGwg YXNrLg0KPj4+Pj4+Pj4gDQo+Pj4+Pj4+PiBJIGFncmVlLCB0aG91Z2gsIHRoYXQgdGltaW5nIG1p Z2h0IGJlIGEgcHJvYmxlbToNCj4+Pj4+Pj4+IHRoZXNlIHN5c3RlbXMgYWxsIGhhdmUgcmVhbCBz ZXJpYWwgY29uc29sZXMgdmlhDQo+Pj4+Pj4+PiBpTE9NLCBzbyAvdi9sL20gdHJhZmZpYyBkb2Vz IGJyaW5nIGV2ZXJ5dGhpbmcgdG8NCj4+Pj4+Pj4+IGEgc3RhbmRzdGlsbC4NCj4+Pj4+Pj4+IA0K Pj4+Pj4+Pj4gTWVhbndoaWxlLCB3aGF0J3MgeW91J3JlIG9waW5pb24gYWJvdXQgQVVUSF9GQUlM RUQ/DQo+Pj4+Pj4+PiBTaG91bGQgdGhlIHNlcnZlciByZXR1cm4gUlBDU0VDX0dTU19DVFhQUk9C TEVNDQo+Pj4+Pj4+PiBpbiB0aGlzIGNhc2UgaW5zdGVhZD8gSWYgaXQgZGlkLCBkbyB5b3UgdGhp bmsNCj4+Pj4+Pj4+IHRoZSBMaW51eCBjbGllbnQgd291bGQgcmVjb3ZlciBieSBjcmVhdGluZyBh DQo+Pj4+Pj4+PiByZXBsYWNlbWVudCBHU1MgY29udGV4dD8NCj4+Pj4+Pj4gDQo+Pj4+Pj4+IEFo LCB5ZXMsIEkgZXF1YXRlZCBBVVRIX0ZBSUxFRCBBbmQgQVVUSF9FUlJPUiBpbiBteSBtaW5kLiBJ ZiBjbGllbnQNCj4+Pj4+Pj4gcmVjZWl2ZXMgdGhlIHJlYXNvbiBhcyBBVVRIX0ZBSUxFRCBhcyBv cHBvc2UgdG8gQ1RYUFJPQkxFTSBpdCB3aWxsDQo+Pj4+Pj4+IGZhaWwgd2l0aCBFSU8gZXJyb3Ig YW5kIHdpbGwgbm90IHRyeSB0byBjcmVhdGUgYSBuZXcgR1NTIGNvbnRleHQuIFNvDQo+Pj4+Pj4+ IHllcywgSSBiZWxpZXZlIGl0IHdvdWxkIGhlbHAgaWYgdGhlIHNlcnZlciByZXR1cm5zIGFueSBv ZiB0aGUNCj4+Pj4+Pj4gZm9sbG93aW5nIGVycm9yczoNCj4+Pj4+Pj4gICAgICAgICAgICAgIGNh c2UgUlBDX0FVVEhfUkVKRUNURURDUkVEOg0KPj4+Pj4+PiAgICAgICAgICAgICAgY2FzZSBSUENf QVVUSF9SRUpFQ1RFRFZFUkY6DQo+Pj4+Pj4+ICAgICAgICAgICAgICBjYXNlIFJQQ1NFQ19HU1Nf Q1JFRFBST0JMRU06DQo+Pj4+Pj4+ICAgICAgICAgICAgICBjYXNlIFJQQ1NFQ19HU1NfQ1RYUFJP QkxFTToNCj4+Pj4+Pj4gDQo+Pj4+Pj4+IHRoZW4gdGhlIGNsaWVudCB3aWxsIHJlY3JlYXRlIHRo ZSBjb250ZXh0Lg0KPj4+Pj4+IA0KPj4+Pj4+IEFsc28gaW4gbXkgdGVzdGluZywgSSBjYW4gc2Vl IHRoYXQgY3JlZGVudGlhbCBjYWNoZSBpcyBwZXIgZ3NzIGZsYXZvci4NCj4+Pj4+PiBKdXN0IHRv IGNoZWNrLCB3aGF0IGtlcm5lbCB2ZXJzaW9uIGlzIHRoaXMgcHJvYmxlbSBlbmNvdW50ZXJlZCBv biAoSQ0KPj4+Pj4+IGtub3cgeW91IHNhaWQgdXBzdHJlYW0pIGJ1dCBJIGp1c3Qgd2FudCB0byBk b3VibGUgY2hlY2sgc28gdGhhdCBJIGNhbg0KPj4+Pj4+IGxvb2sgYXQgdGhlIGNvcnJlY3Qgc291 cmNlIGNvZGUuDQo+Pj4+PiANCj4+Pj4+IHY0LjEuMTIgKHN0YWJsZSkgSSB0aGluay4NCj4+Pj4g DQo+Pj4+IEFsc28sIGNhbiB5b3Ugc2hhcmUgdGhlIG5ldHdvcmsgdHJhY2U/DQo+Pj4gDQo+Pj4g SGkgQ2h1Y2ssDQo+Pj4gDQo+Pj4gSSB3YXMgZmluYWxseSBhYmxlIHRvIHJlcHJvZHVjZSB0aGUg Y29uZGl0aW9uIHlvdSB3ZXJlIHNlZWluZyAoaS5lLiwNCj4+PiB0aGUgdXNlIG9mIHRoZSBzYW1l IGNvbnRleHQgZm9yIGRpZmZlcmVudCBnc3Mgc2VydmljZXMpLg0KPj4+IA0KPj4+IEkgZW5hYmxl ZCBycGNkZWJ1ZyBycGMgYXV0aCBhbmQgSSBjYW4gc2VlIHRoYXQgdGhlIDJuZCByZXF1ZXN0IGVu ZHMgdXANCj4+PiBmaW5kaW5nIGEgZ3NzX3VwY2FsbCBtZXNzYWdlIGJlY2F1c2UgaXQncyBqdXN0 IG1hdGNoZWQgYnkgdGhlIHVpZC4NCj4+PiBUaGVyZSBpcyBldmVuIGEgY29tbWVudCBpbiBhdXRo X2dzcy9hdXRoX2dzcy5jIGluIGdzc19hZGRfbXNnKCkgc2F5aW5nDQo+Pj4gdGhhdCBpZiB0aGVy ZSBpcyB1cGNhbGwgZm9yIGFuIHVpZCB0aGVuIGl0IHdvbid0IGFkZCBhbm90aGVyIHVwY2FsbC4N Cj4+PiBTbyBJIHRoaW5rIHRoZSBkZWNpc2lvbiBpcyBtYWRlIHJpZ2h0IHRoZXJlIHRvIHNoYXJl IHRoZSBzYW1lIGNvbnRleHQNCj4+PiBubyBtYXR0ZXIgdGhlIGdzcyBzZXJ2aWNlLg0KPj4gDQo+ PiBJZiBJIHVuZGVyc3RhbmQgY29ycmVjdGx5LCB0aGF0J3MganVzdCB3aGF0IEFuZHkgcHJlZGlj dGVkLg0KPj4gDQo+PiBUaGF0IGNoZWNrIG5lZWRzIHRvIGJlIGNoYW5nZWQgdG8gYWxsb3cgYW5v dGhlciB1cGNhbGwgdG8gYmUNCj4+IHF1ZXVlZCBpZiB0aGUgVUlEIG1hdGNoZXMgYnV0IHRoZSBH U1Mgc2VydmljZSBkb2VzIG5vdC4NCj4gDQo+IFlvdSBzaG91bGQgYmUgYWJsZSB0byB1c2UgdGhl IHNhbWUgY29udGV4dCB3aXRoIGRpZmZlcmVudCBzZXJ2aWNlcy4NCj4gDQo+IEFwb2xvZ2llcywg SSBoYXZlbid0IGNhdWdodCB1cCB3aXRoIHRoZSB3aG9sZSBkaXNjdXNzaW9uIGFib3ZlLCB0aGlz IG9uZQ0KPiBwb2ludCBqdXN0IGp1bXBlZCBvdXQgYXQgbWUuICBJZiB5b3UncmUgdHJ5aW5nIHRv IHJlcXVlc3QgYSB3aG9sZSBuZXcNCj4gZ3NzIGNvbnRleHQganVzdCBzbyB5b3UgY2FuIHVzZSwg ZS5nLiwgaW50ZWdyaXR5IGluc3RlYWQgb2YgcHJpdmFjeSwNCj4gdGhlbiBzb21ldGhpbmcncyB3 cm9uZy4NCg0KVGhlIGNsaWVudCBjb2RlIGhhcyBzZXBhcmF0ZSBnc3NfY3JlZCBjYWNoZXMgKGFu ZCBzbyBzZXBhcmF0ZSBnc3NfY29udGV44oCZcykgcGVyIGdzc19hdXRoLCB3aGljaCBpcyBwZXIg c2VydmljZS4gQUZBSUsgdGhlIGNsaWVudCBoYXMgYWx3YXlzIG9idGFpbmVkIGEgc2VwYXJhdGUg Y29udGV4dCBwZXIgc2VydmljZSBwZXIgc2VydmVyLiBXaGlsZSB3ZSBjYW4gdXNlIHRoZSBzYW1l IGdzcyBjb250ZXh0IHdpdGggZGlmZmVyZW50IHNlcnZpY2VzLCB0aGF0IGlzIG5vdCB0aGUgZGVz aWduIGNob2ljZS4NCg0K4oCUPkFuZHkNCg0KDQo+IA0KPiAtLWIuDQoNCg== ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 18:53 ` Adamson, Andy @ 2016-08-03 19:56 ` Olga Kornievskaia 2016-08-03 20:06 ` J. Bruce Fields 0 siblings, 1 reply; 25+ messages in thread From: Olga Kornievskaia @ 2016-08-03 19:56 UTC (permalink / raw) To: Adamson, Andy; +Cc: J. Bruce Fields, Chuck Lever, Linux NFS Mailing List On Wed, Aug 3, 2016 at 2:53 PM, Adamson, Andy <William.Adamson@netapp.com> wrote: > >> On Aug 2, 2016, at 2:06 PM, J. Bruce Fields <bfields@fieldses.org> wrote: >> >> On Fri, Jul 29, 2016 at 12:38:34PM -0400, Chuck Lever wrote: >>> >>>> On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> >>>> On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>> >>>>>> >>>>>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>> >>>>>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>> >>>>>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>>> >>>>>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>>>>>>>> <William.Adamson@netapp.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Andy- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>>>>>>>> report. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>>>>>>>> and NFSv4.1 mount points. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>>>>>>>> the network capture. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>>>>>>>> >>>>>>>>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>>>>>>>> lease management operations, which is bound to the client's >>>>>>>>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>>>>>>>> Call this GSS context 1. >>>>>>>>>>>>>> >>>>>>>>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>>>>>>>> established for NFS operations associated with that user. >>>>>>>>>>>>>> Since the reproducer is running as root, this context is >>>>>>>>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>>>>>>>> >>>>>>>>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>>>>>>>> connection with the server, and performs a RENEW >>>>>>>>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>>>>>>>> problem at all). >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>>>>>>>> service "none" is used for this operation, since it is on >>>>>>>>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The RPC is accepted, but the server reports >>>>>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client responds by attempting state recovery. The >>>>>>>>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>>>>>>>> credential cache again and finds the recently established >>>>>>>>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>>>>>>>> 3 with GSS service "integrity." >>>>>>>>>>>>>> >>>>>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>>>>>>>> the client reports that "check lease failed" and >>>>>>>>>>>>>> terminates state recovery. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>>>>>>>> stateid with predictable results. The client again tries >>>>>>>>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>>>>>>>> management operations, attempt to use the same GSS context >>>>>>>>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>>>>>>>> services. >>>>>>>>>>>>> >>>>>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> The key issue seems to be why, when the mount >>>>>>>>>>>>>> is first established, the client is correctly able to >>>>>>>>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>>>>>>>> context with two different GSS services. >>>>>>>>>>>>> >>>>>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>>>>>>>> >>>>>>>>>>>> I agree with Andy. It must be a tight race. >>>>>>>>>>> >>>>>>>>>>> In one capture I see something like this after >>>>>>>>>>> the server restarts: >>>>>>>>>>> >>>>>>>>>>> SYN >>>>>>>>>>> SYN, ACK >>>>>>>>>>> ACK >>>>>>>>>>> C WRITE >>>>>>>>>>> C SEQUENCE >>>>>>>>>>> R WRITE -> CTX_PROBLEM >>>>>>>>>>> R SEQUENCE -> CTX_PROBLEM >>>>>>>>>>> C NULL (KRB5_AP_REQ) >>>>>>>>>>> R NULL (KRB5_AP_REP) >>>>>>>>>>> C WRITE >>>>>>>>>>> C SEQUENCE >>>>>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>>>>>>>> R SEQUENCE -> AUTH_FAILED >>>>>>>>>>> >>>>>>>>>>> Andy's theory neatly explains this behavior. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> I have tried to reproduce >>>>>>>>>>>> your scenario and in my tests of rebooting the server all recover >>>>>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>>>>>>>> the new context is established and then RENEW using integrity service >>>>>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>>>>>>>> >>>>>>>>>>> If one operation is allowed to complete, then >>>>>>>>>>> the other will always recognize that another >>>>>>>>>>> fresh GSS context is needed. If two are sent >>>>>>>>>>> at the same time, they race and one always >>>>>>>>>>> fails. >>>>>>>>>>> >>>>>>>>>>> Helen's test includes a second idle mount point >>>>>>>>>>> (sec=krb5i) and maybe that is needed to trigger >>>>>>>>>>> the race? >>>>>>>>>> >>>>>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>>>>>>>> failure (gssd optionally) (i realize that it might alter the timings >>>>>>>>>> and not hit the issue but worth a shot)? >>>>>>>>> >>>>>>>>> I'm sure that's fine. An internal tester hit this, >>>>>>>>> not a customer, so I will ask. >>>>>>>>> >>>>>>>>> I agree, though, that timing might be a problem: >>>>>>>>> these systems all have real serial consoles via >>>>>>>>> iLOM, so /v/l/m traffic does bring everything to >>>>>>>>> a standstill. >>>>>>>>> >>>>>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? >>>>>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM >>>>>>>>> in this case instead? If it did, do you think >>>>>>>>> the Linux client would recover by creating a >>>>>>>>> replacement GSS context? >>>>>>>> >>>>>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >>>>>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >>>>>>>> fail with EIO error and will not try to create a new GSS context. So >>>>>>>> yes, I believe it would help if the server returns any of the >>>>>>>> following errors: >>>>>>>> case RPC_AUTH_REJECTEDCRED: >>>>>>>> case RPC_AUTH_REJECTEDVERF: >>>>>>>> case RPCSEC_GSS_CREDPROBLEM: >>>>>>>> case RPCSEC_GSS_CTXPROBLEM: >>>>>>>> >>>>>>>> then the client will recreate the context. >>>>>>> >>>>>>> Also in my testing, I can see that credential cache is per gss flavor. >>>>>>> Just to check, what kernel version is this problem encountered on (I >>>>>>> know you said upstream) but I just want to double check so that I can >>>>>>> look at the correct source code. >>>>>> >>>>>> v4.1.12 (stable) I think. >>>>> >>>>> Also, can you share the network trace? >>>> >>>> Hi Chuck, >>>> >>>> I was finally able to reproduce the condition you were seeing (i.e., >>>> the use of the same context for different gss services). >>>> >>>> I enabled rpcdebug rpc auth and I can see that the 2nd request ends up >>>> finding a gss_upcall message because it's just matched by the uid. >>>> There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying >>>> that if there is upcall for an uid then it won't add another upcall. >>>> So I think the decision is made right there to share the same context >>>> no matter the gss service. >>> >>> If I understand correctly, that's just what Andy predicted. >>> >>> That check needs to be changed to allow another upcall to be >>> queued if the UID matches but the GSS service does not. >> >> You should be able to use the same context with different services. >> >> Apologies, I haven't caught up with the whole discussion above, this one >> point just jumped out at me. If you're trying to request a whole new >> gss context just so you can use, e.g., integrity instead of privacy, >> then something's wrong. > > The client code has separate gss_cred caches (and so separate gss_contex’s) per gss_auth, which is per service. AFAIK the client has always obtained a separate context per service per server. While we can use the same gss context with different services, that is not the design choice. Given the current code, I'd say that it's not clear what the design choice is. The upcall code states that it will not do another upcall for a given UID if another upcall is already made. So it's purely timing luck that we have two different contexts for the mount when sec=krb5 and the default for lease operations is "integrity". And we don't pass the GSS service between the kernel and gssd and thus no way for us to tie the upcall to the service. > > —>Andy > > >> >> --b. > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 19:56 ` Olga Kornievskaia @ 2016-08-03 20:06 ` J. Bruce Fields 2016-08-03 20:11 ` Olga Kornievskaia 2016-08-03 20:18 ` Adamson, Andy 0 siblings, 2 replies; 25+ messages in thread From: J. Bruce Fields @ 2016-08-03 20:06 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: Adamson, Andy, Chuck Lever, Linux NFS Mailing List On Wed, Aug 03, 2016 at 03:56:31PM -0400, Olga Kornievskaia wrote: > On Wed, Aug 3, 2016 at 2:53 PM, Adamson, Andy > <William.Adamson@netapp.com> wrote: > > > >> On Aug 2, 2016, at 2:06 PM, J. Bruce Fields <bfields@fieldses.org> wrote: > >> > >> On Fri, Jul 29, 2016 at 12:38:34PM -0400, Chuck Lever wrote: > >>> > >>>> On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>> > >>>> On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>> > >>>>>> > >>>>>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>>> > >>>>>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>>>>> > >>>>>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>>>>>> > >>>>>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy > >>>>>>>>>>>> <William.Adamson@netapp.com> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Andy- > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks for taking the time to discuss this with me. I've > >>>>>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug > >>>>>>>>>>>>>> report. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts > >>>>>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 > >>>>>>>>>>>>>> and NFSv4.1 mount points. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The reproducer is a simple program that generates one NFS > >>>>>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots > >>>>>>>>>>>>>> (or one cluster head fails over to the other and back). The > >>>>>>>>>>>>>> goal of the reproducer is to identify problems with state > >>>>>>>>>>>>>> recovery without a lot of other I/O going on to clutter up > >>>>>>>>>>>>>> the network capture. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount > >>>>>>>>>>>>>> point, and the reproducer is run as root. We've found this > >>>>>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> At mount time, the client establishes a GSS context for > >>>>>>>>>>>>>> lease management operations, which is bound to the client's > >>>>>>>>>>>>>> NFS service principal and uses GSS service "integrity." > >>>>>>>>>>>>>> Call this GSS context 1. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> When the reproducer starts, a second GSS context is > >>>>>>>>>>>>>> established for NFS operations associated with that user. > >>>>>>>>>>>>>> Since the reproducer is running as root, this context is > >>>>>>>>>>>>>> also bound to the client's NFS service principal, but it > >>>>>>>>>>>>>> uses the GSS service "none" (reflecting the explicit > >>>>>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> After the server reboots, the client re-establishes a TCP > >>>>>>>>>>>>>> connection with the server, and performs a RENEW > >>>>>>>>>>>>>> operation using context 1. Thanks to the server reboot, > >>>>>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects > >>>>>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 > >>>>>>>>>>>>>> NULL operation. Call this GSS context 3. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Interestingly, the client does not resend the RENEW > >>>>>>>>>>>>>> operation at this point (if it did, we wouldn't see this > >>>>>>>>>>>>>> problem at all). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The client then attempts to resume the reproducer workload. > >>>>>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available > >>>>>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, > >>>>>>>>>>>>>> already bound to the client's NFS service principal. But GSS > >>>>>>>>>>>>>> service "none" is used for this operation, since it is on > >>>>>>>>>>>>>> behalf of the mount where sec=krb5 was specified. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The RPC is accepted, but the server reports > >>>>>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The client responds by attempting state recovery. The > >>>>>>>>>>>>>> first operation it tries is another RENEW. Since this is > >>>>>>>>>>>>>> a lease management operation, the client looks in UID 0's > >>>>>>>>>>>>>> credential cache again and finds the recently established > >>>>>>>>>>>>>> context 3. It tries the RENEW operation using GSS context > >>>>>>>>>>>>>> 3 with GSS service "integrity." > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and > >>>>>>>>>>>>>> the client reports that "check lease failed" and > >>>>>>>>>>>>>> terminates state recovery. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The client re-drives the WRITE operation with the stale > >>>>>>>>>>>>>> stateid with predictable results. The client again tries > >>>>>>>>>>>>>> to recover state by sending a RENEW, and still uses the > >>>>>>>>>>>>>> same GSS context 3 with service "integrity" and gets the > >>>>>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop > >>>>>>>>>>>>>> ensues, and the client mount point is deadlocked. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Your analysis was that because the reproducer is run as > >>>>>>>>>>>>>> root, both the reproducer's I/O operations, and lease > >>>>>>>>>>>>>> management operations, attempt to use the same GSS context > >>>>>>>>>>>>>> in UID 0's credential cache, but each uses different GSS > >>>>>>>>>>>>>> services. > >>>>>>>>>>>>> > >>>>>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” > >>>>>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. > >>>>>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> The key issue seems to be why, when the mount > >>>>>>>>>>>>>> is first established, the client is correctly able to > >>>>>>>>>>>>>> establish two separate GSS contexts for UID 0; but after > >>>>>>>>>>>>>> a server reboot, the client attempts to use the same GSS > >>>>>>>>>>>>>> context with two different GSS services. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. > >>>>>>>>>>>> > >>>>>>>>>>>> I agree with Andy. It must be a tight race. > >>>>>>>>>>> > >>>>>>>>>>> In one capture I see something like this after > >>>>>>>>>>> the server restarts: > >>>>>>>>>>> > >>>>>>>>>>> SYN > >>>>>>>>>>> SYN, ACK > >>>>>>>>>>> ACK > >>>>>>>>>>> C WRITE > >>>>>>>>>>> C SEQUENCE > >>>>>>>>>>> R WRITE -> CTX_PROBLEM > >>>>>>>>>>> R SEQUENCE -> CTX_PROBLEM > >>>>>>>>>>> C NULL (KRB5_AP_REQ) > >>>>>>>>>>> R NULL (KRB5_AP_REP) > >>>>>>>>>>> C WRITE > >>>>>>>>>>> C SEQUENCE > >>>>>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID > >>>>>>>>>>> R SEQUENCE -> AUTH_FAILED > >>>>>>>>>>> > >>>>>>>>>>> Andy's theory neatly explains this behavior. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> I have tried to reproduce > >>>>>>>>>>>> your scenario and in my tests of rebooting the server all recover > >>>>>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then > >>>>>>>>>>>> the new context is established and then RENEW using integrity service > >>>>>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which > >>>>>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) > >>>>>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none > >>>>>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different > >>>>>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and > >>>>>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. > >>>>>>>>>>> > >>>>>>>>>>> If one operation is allowed to complete, then > >>>>>>>>>>> the other will always recognize that another > >>>>>>>>>>> fresh GSS context is needed. If two are sent > >>>>>>>>>>> at the same time, they race and one always > >>>>>>>>>>> fails. > >>>>>>>>>>> > >>>>>>>>>>> Helen's test includes a second idle mount point > >>>>>>>>>>> (sec=krb5i) and maybe that is needed to trigger > >>>>>>>>>>> the race? > >>>>>>>>>> > >>>>>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the > >>>>>>>>>> failure (gssd optionally) (i realize that it might alter the timings > >>>>>>>>>> and not hit the issue but worth a shot)? > >>>>>>>>> > >>>>>>>>> I'm sure that's fine. An internal tester hit this, > >>>>>>>>> not a customer, so I will ask. > >>>>>>>>> > >>>>>>>>> I agree, though, that timing might be a problem: > >>>>>>>>> these systems all have real serial consoles via > >>>>>>>>> iLOM, so /v/l/m traffic does bring everything to > >>>>>>>>> a standstill. > >>>>>>>>> > >>>>>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? > >>>>>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM > >>>>>>>>> in this case instead? If it did, do you think > >>>>>>>>> the Linux client would recover by creating a > >>>>>>>>> replacement GSS context? > >>>>>>>> > >>>>>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client > >>>>>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will > >>>>>>>> fail with EIO error and will not try to create a new GSS context. So > >>>>>>>> yes, I believe it would help if the server returns any of the > >>>>>>>> following errors: > >>>>>>>> case RPC_AUTH_REJECTEDCRED: > >>>>>>>> case RPC_AUTH_REJECTEDVERF: > >>>>>>>> case RPCSEC_GSS_CREDPROBLEM: > >>>>>>>> case RPCSEC_GSS_CTXPROBLEM: > >>>>>>>> > >>>>>>>> then the client will recreate the context. > >>>>>>> > >>>>>>> Also in my testing, I can see that credential cache is per gss flavor. > >>>>>>> Just to check, what kernel version is this problem encountered on (I > >>>>>>> know you said upstream) but I just want to double check so that I can > >>>>>>> look at the correct source code. > >>>>>> > >>>>>> v4.1.12 (stable) I think. > >>>>> > >>>>> Also, can you share the network trace? > >>>> > >>>> Hi Chuck, > >>>> > >>>> I was finally able to reproduce the condition you were seeing (i.e., > >>>> the use of the same context for different gss services). > >>>> > >>>> I enabled rpcdebug rpc auth and I can see that the 2nd request ends up > >>>> finding a gss_upcall message because it's just matched by the uid. > >>>> There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying > >>>> that if there is upcall for an uid then it won't add another upcall. > >>>> So I think the decision is made right there to share the same context > >>>> no matter the gss service. > >>> > >>> If I understand correctly, that's just what Andy predicted. > >>> > >>> That check needs to be changed to allow another upcall to be > >>> queued if the UID matches but the GSS service does not. > >> > >> You should be able to use the same context with different services. > >> > >> Apologies, I haven't caught up with the whole discussion above, this one > >> point just jumped out at me. If you're trying to request a whole new > >> gss context just so you can use, e.g., integrity instead of privacy, > >> then something's wrong. > > > > The client code has separate gss_cred caches (and so separate gss_contex’s) per gss_auth, which is per service. AFAIK the client has always obtained a separate context per service per server. While we can use the same gss context with different services, that is not the design choice. > > Given the current code, I'd say that it's not clear what the design > choice is. The upcall code states that it will not do another upcall > for a given UID if another upcall is already made. So it's purely > timing luck that we have two different contexts for the mount when > sec=krb5 and the default for lease operations is "integrity". And we > don't pass the GSS service between the kernel and gssd and thus no way > for us to tie the upcall to the service. Well, I wrote some of the upcall code, and I though (incorrectly, I guess) that clients could reuse the same context with multiple services, so the confusion may just be my fault. So you think the only fix here is to key the upcalls on (mechanism, service, uid) instead of just (mechanism, uid)? --b. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 20:06 ` J. Bruce Fields @ 2016-08-03 20:11 ` Olga Kornievskaia 2016-08-03 20:18 ` Adamson, Andy 1 sibling, 0 replies; 25+ messages in thread From: Olga Kornievskaia @ 2016-08-03 20:11 UTC (permalink / raw) To: J. Bruce Fields; +Cc: Adamson, Andy, Chuck Lever, Linux NFS Mailing List On Wed, Aug 3, 2016 at 4:06 PM, J. Bruce Fields <bfields@fieldses.org> wrote: > On Wed, Aug 03, 2016 at 03:56:31PM -0400, Olga Kornievskaia wrote: >> On Wed, Aug 3, 2016 at 2:53 PM, Adamson, Andy >> <William.Adamson@netapp.com> wrote: >> > >> >> On Aug 2, 2016, at 2:06 PM, J. Bruce Fields <bfields@fieldses.org> wrote: >> >> >> >> On Fri, Jul 29, 2016 at 12:38:34PM -0400, Chuck Lever wrote: >> >>> >> >>>> On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>>> >> >>>> On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>>>> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >>>>>> >> >>>>>> >> >>>>>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>>>>>> >> >>>>>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>>>>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >>>>>>>>> >> >>>>>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>>>>>>>>> >> >>>>>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >> >>>>>>>>>>>> <William.Adamson@netapp.com> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hi Andy- >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks for taking the time to discuss this with me. I've >> >>>>>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >> >>>>>>>>>>>>>> report. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts >> >>>>>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >> >>>>>>>>>>>>>> and NFSv4.1 mount points. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The reproducer is a simple program that generates one NFS >> >>>>>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots >> >>>>>>>>>>>>>> (or one cluster head fails over to the other and back). The >> >>>>>>>>>>>>>> goal of the reproducer is to identify problems with state >> >>>>>>>>>>>>>> recovery without a lot of other I/O going on to clutter up >> >>>>>>>>>>>>>> the network capture. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount >> >>>>>>>>>>>>>> point, and the reproducer is run as root. We've found this >> >>>>>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> At mount time, the client establishes a GSS context for >> >>>>>>>>>>>>>> lease management operations, which is bound to the client's >> >>>>>>>>>>>>>> NFS service principal and uses GSS service "integrity." >> >>>>>>>>>>>>>> Call this GSS context 1. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> When the reproducer starts, a second GSS context is >> >>>>>>>>>>>>>> established for NFS operations associated with that user. >> >>>>>>>>>>>>>> Since the reproducer is running as root, this context is >> >>>>>>>>>>>>>> also bound to the client's NFS service principal, but it >> >>>>>>>>>>>>>> uses the GSS service "none" (reflecting the explicit >> >>>>>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> After the server reboots, the client re-establishes a TCP >> >>>>>>>>>>>>>> connection with the server, and performs a RENEW >> >>>>>>>>>>>>>> operation using context 1. Thanks to the server reboot, >> >>>>>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >> >>>>>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >> >>>>>>>>>>>>>> NULL operation. Call this GSS context 3. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Interestingly, the client does not resend the RENEW >> >>>>>>>>>>>>>> operation at this point (if it did, we wouldn't see this >> >>>>>>>>>>>>>> problem at all). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The client then attempts to resume the reproducer workload. >> >>>>>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >> >>>>>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >> >>>>>>>>>>>>>> already bound to the client's NFS service principal. But GSS >> >>>>>>>>>>>>>> service "none" is used for this operation, since it is on >> >>>>>>>>>>>>>> behalf of the mount where sec=krb5 was specified. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The RPC is accepted, but the server reports >> >>>>>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The client responds by attempting state recovery. The >> >>>>>>>>>>>>>> first operation it tries is another RENEW. Since this is >> >>>>>>>>>>>>>> a lease management operation, the client looks in UID 0's >> >>>>>>>>>>>>>> credential cache again and finds the recently established >> >>>>>>>>>>>>>> context 3. It tries the RENEW operation using GSS context >> >>>>>>>>>>>>>> 3 with GSS service "integrity." >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >> >>>>>>>>>>>>>> the client reports that "check lease failed" and >> >>>>>>>>>>>>>> terminates state recovery. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The client re-drives the WRITE operation with the stale >> >>>>>>>>>>>>>> stateid with predictable results. The client again tries >> >>>>>>>>>>>>>> to recover state by sending a RENEW, and still uses the >> >>>>>>>>>>>>>> same GSS context 3 with service "integrity" and gets the >> >>>>>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >> >>>>>>>>>>>>>> ensues, and the client mount point is deadlocked. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Your analysis was that because the reproducer is run as >> >>>>>>>>>>>>>> root, both the reproducer's I/O operations, and lease >> >>>>>>>>>>>>>> management operations, attempt to use the same GSS context >> >>>>>>>>>>>>>> in UID 0's credential cache, but each uses different GSS >> >>>>>>>>>>>>>> services. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >> >>>>>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >> >>>>>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> The key issue seems to be why, when the mount >> >>>>>>>>>>>>>> is first established, the client is correctly able to >> >>>>>>>>>>>>>> establish two separate GSS contexts for UID 0; but after >> >>>>>>>>>>>>>> a server reboot, the client attempts to use the same GSS >> >>>>>>>>>>>>>> context with two different GSS services. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >> >>>>>>>>>>>> >> >>>>>>>>>>>> I agree with Andy. It must be a tight race. >> >>>>>>>>>>> >> >>>>>>>>>>> In one capture I see something like this after >> >>>>>>>>>>> the server restarts: >> >>>>>>>>>>> >> >>>>>>>>>>> SYN >> >>>>>>>>>>> SYN, ACK >> >>>>>>>>>>> ACK >> >>>>>>>>>>> C WRITE >> >>>>>>>>>>> C SEQUENCE >> >>>>>>>>>>> R WRITE -> CTX_PROBLEM >> >>>>>>>>>>> R SEQUENCE -> CTX_PROBLEM >> >>>>>>>>>>> C NULL (KRB5_AP_REQ) >> >>>>>>>>>>> R NULL (KRB5_AP_REP) >> >>>>>>>>>>> C WRITE >> >>>>>>>>>>> C SEQUENCE >> >>>>>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID >> >>>>>>>>>>> R SEQUENCE -> AUTH_FAILED >> >>>>>>>>>>> >> >>>>>>>>>>> Andy's theory neatly explains this behavior. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>>> I have tried to reproduce >> >>>>>>>>>>>> your scenario and in my tests of rebooting the server all recover >> >>>>>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >> >>>>>>>>>>>> the new context is established and then RENEW using integrity service >> >>>>>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >> >>>>>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >> >>>>>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >> >>>>>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >> >>>>>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and >> >>>>>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >> >>>>>>>>>>> >> >>>>>>>>>>> If one operation is allowed to complete, then >> >>>>>>>>>>> the other will always recognize that another >> >>>>>>>>>>> fresh GSS context is needed. If two are sent >> >>>>>>>>>>> at the same time, they race and one always >> >>>>>>>>>>> fails. >> >>>>>>>>>>> >> >>>>>>>>>>> Helen's test includes a second idle mount point >> >>>>>>>>>>> (sec=krb5i) and maybe that is needed to trigger >> >>>>>>>>>>> the race? >> >>>>>>>>>> >> >>>>>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >> >>>>>>>>>> failure (gssd optionally) (i realize that it might alter the timings >> >>>>>>>>>> and not hit the issue but worth a shot)? >> >>>>>>>>> >> >>>>>>>>> I'm sure that's fine. An internal tester hit this, >> >>>>>>>>> not a customer, so I will ask. >> >>>>>>>>> >> >>>>>>>>> I agree, though, that timing might be a problem: >> >>>>>>>>> these systems all have real serial consoles via >> >>>>>>>>> iLOM, so /v/l/m traffic does bring everything to >> >>>>>>>>> a standstill. >> >>>>>>>>> >> >>>>>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? >> >>>>>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM >> >>>>>>>>> in this case instead? If it did, do you think >> >>>>>>>>> the Linux client would recover by creating a >> >>>>>>>>> replacement GSS context? >> >>>>>>>> >> >>>>>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >> >>>>>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >> >>>>>>>> fail with EIO error and will not try to create a new GSS context. So >> >>>>>>>> yes, I believe it would help if the server returns any of the >> >>>>>>>> following errors: >> >>>>>>>> case RPC_AUTH_REJECTEDCRED: >> >>>>>>>> case RPC_AUTH_REJECTEDVERF: >> >>>>>>>> case RPCSEC_GSS_CREDPROBLEM: >> >>>>>>>> case RPCSEC_GSS_CTXPROBLEM: >> >>>>>>>> >> >>>>>>>> then the client will recreate the context. >> >>>>>>> >> >>>>>>> Also in my testing, I can see that credential cache is per gss flavor. >> >>>>>>> Just to check, what kernel version is this problem encountered on (I >> >>>>>>> know you said upstream) but I just want to double check so that I can >> >>>>>>> look at the correct source code. >> >>>>>> >> >>>>>> v4.1.12 (stable) I think. >> >>>>> >> >>>>> Also, can you share the network trace? >> >>>> >> >>>> Hi Chuck, >> >>>> >> >>>> I was finally able to reproduce the condition you were seeing (i.e., >> >>>> the use of the same context for different gss services). >> >>>> >> >>>> I enabled rpcdebug rpc auth and I can see that the 2nd request ends up >> >>>> finding a gss_upcall message because it's just matched by the uid. >> >>>> There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying >> >>>> that if there is upcall for an uid then it won't add another upcall. >> >>>> So I think the decision is made right there to share the same context >> >>>> no matter the gss service. >> >>> >> >>> If I understand correctly, that's just what Andy predicted. >> >>> >> >>> That check needs to be changed to allow another upcall to be >> >>> queued if the UID matches but the GSS service does not. >> >> >> >> You should be able to use the same context with different services. >> >> >> >> Apologies, I haven't caught up with the whole discussion above, this one >> >> point just jumped out at me. If you're trying to request a whole new >> >> gss context just so you can use, e.g., integrity instead of privacy, >> >> then something's wrong. >> > >> > The client code has separate gss_cred caches (and so separate gss_contex’s) per gss_auth, which is per service. AFAIK the client has always obtained a separate context per service per server. While we can use the same gss context with different services, that is not the design choice. >> >> Given the current code, I'd say that it's not clear what the design >> choice is. The upcall code states that it will not do another upcall >> for a given UID if another upcall is already made. So it's purely >> timing luck that we have two different contexts for the mount when >> sec=krb5 and the default for lease operations is "integrity". And we >> don't pass the GSS service between the kernel and gssd and thus no way >> for us to tie the upcall to the service. > > Well, I wrote some of the upcall code, and I though (incorrectly, I > guess) that clients could reuse the same context with multiple services, > so the confusion may just be my fault. So you think the only fix here > is to key the upcalls on (mechanism, service, uid) instead of just > (mechanism, uid)? Right now I'm going to check on the 'calling to gssd' if the existing upcall has the same UIDs but not the same gss service, then still place another upcall. But the lookup from the gssd is just based on the UID caz that's all we have. Maybe that'll work? > > --b. > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 20:06 ` J. Bruce Fields 2016-08-03 20:11 ` Olga Kornievskaia @ 2016-08-03 20:18 ` Adamson, Andy 2016-08-03 20:33 ` Trond Myklebust 1 sibling, 1 reply; 25+ messages in thread From: Adamson, Andy @ 2016-08-03 20:18 UTC (permalink / raw) To: J. Bruce Fields Cc: Olga Kornievskaia, Adamson, Andy, Chuck Lever, Linux NFS Mailing List DQo+IE9uIEF1ZyAzLCAyMDE2LCBhdCA0OjA2IFBNLCBKLiBCcnVjZSBGaWVsZHMgPGJmaWVsZHNA ZmllbGRzZXMub3JnPiB3cm90ZToNCj4gDQo+IE9uIFdlZCwgQXVnIDAzLCAyMDE2IGF0IDAzOjU2 OjMxUE0gLTA0MDAsIE9sZ2EgS29ybmlldnNrYWlhIHdyb3RlOg0KPj4gT24gV2VkLCBBdWcgMywg MjAxNiBhdCAyOjUzIFBNLCBBZGFtc29uLCBBbmR5DQo+PiA8V2lsbGlhbS5BZGFtc29uQG5ldGFw cC5jb20+IHdyb3RlOg0KPj4+IA0KPj4+PiBPbiBBdWcgMiwgMjAxNiwgYXQgMjowNiBQTSwgSi4g QnJ1Y2UgRmllbGRzIDxiZmllbGRzQGZpZWxkc2VzLm9yZz4gd3JvdGU6DQo+Pj4+IA0KPj4+PiBP biBGcmksIEp1bCAyOSwgMjAxNiBhdCAxMjozODozNFBNIC0wNDAwLCBDaHVjayBMZXZlciB3cm90 ZToNCj4+Pj4+IA0KPj4+Pj4+IE9uIEp1bCAyOSwgMjAxNiwgYXQgMTI6MjcgUE0sIE9sZ2EgS29y bmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4gDQo+Pj4+Pj4gT24gTW9u LCBKdWwgMjUsIDIwMTYgYXQgMjoxOCBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2gu ZWR1PiB3cm90ZToNCj4+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgNTozMiBQTSwgQ2h1 Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4gDQo+Pj4+ Pj4+PiANCj4+Pj4+Pj4+PiBPbiBKdWwgMjEsIDIwMTYsIGF0IDEwOjQ2IFBNLCBPbGdhIEtvcm5p ZXZza2FpYSA8YWdsb0B1bWljaC5lZHU+IHdyb3RlOg0KPj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBP biBUaHUsIEp1bCAyMSwgMjAxNiBhdCAzOjU0IFBNLCBPbGdhIEtvcm5pZXZza2FpYSA8YWdsb0B1 bWljaC5lZHU+IHdyb3RlOg0KPj4+Pj4+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgMTo1 NiBQTSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+ Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gT24gSnVsIDIxLCAyMDE2LCBhdCA2OjA0IFBNLCBPbGdhIEtv cm5pZXZza2FpYSA8YWdsb0B1bWljaC5lZHU+IHdyb3RlOg0KPj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+ Pj4+Pj4+IE9uIFRodSwgSnVsIDIxLCAyMDE2IGF0IDI6NTUgQU0sIENodWNrIExldmVyIDxjaHVj ay5sZXZlckBvcmFjbGUuY29tPiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+ PiBPbiBKdWwgMjAsIDIwMTYsIGF0IDY6NTYgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVt aWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+IE9uIFdlZCwg SnVsIDIwLCAyMDE2IGF0IDU6MTQgQU0sIEFkYW1zb24sIEFuZHkNCj4+Pj4+Pj4+Pj4+Pj4+IDxX aWxsaWFtLkFkYW1zb25AbmV0YXBwLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+ Pj4+Pj4+Pj4+Pj4+IE9uIEp1bCAxOSwgMjAxNiwgYXQgMTA6NTEgQU0sIENodWNrIExldmVyIDxj aHVjay5sZXZlckBvcmFjbGUuY29tPiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+ Pj4+Pj4+Pj4+IEhpIEFuZHktDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+PiBU aGFua3MgZm9yIHRha2luZyB0aGUgdGltZSB0byBkaXNjdXNzIHRoaXMgd2l0aCBtZS4gSSd2ZQ0K Pj4+Pj4+Pj4+Pj4+Pj4+PiBjb3BpZWQgbGludXgtbmZzIHRvIG1ha2UgdGhpcyBlLW1haWwgYWxz byBhbiB1cHN0cmVhbSBidWcNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gcmVwb3J0Lg0KPj4+Pj4+Pj4+Pj4+ Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4gQXMgd2Ugc2F3IGluIHRoZSBuZXR3b3JrIGNhcHR1cmUs IHJlY292ZXJ5IG9mIEdTUyBjb250ZXh0cw0KPj4+Pj4+Pj4+Pj4+Pj4+PiBhZnRlciBhIHNlcnZl ciByZWJvb3QgZmFpbHMgaW4gY2VydGFpbiBjYXNlcyB3aXRoIE5GU3Y0LjANCj4+Pj4+Pj4+Pj4+ Pj4+Pj4gYW5kIE5GU3Y0LjEgbW91bnQgcG9pbnRzLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+ Pj4+Pj4+Pj4+Pj4gVGhlIHJlcHJvZHVjZXIgaXMgYSBzaW1wbGUgcHJvZ3JhbSB0aGF0IGdlbmVy YXRlcyBvbmUgTkZTDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IFdSSVRFIHBlcmlvZGljYWxseSwgcnVuIHdo aWxlIHRoZSBzZXJ2ZXIgcmVwZWF0ZWRseSByZWJvb3RzDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IChvciBv bmUgY2x1c3RlciBoZWFkIGZhaWxzIG92ZXIgdG8gdGhlIG90aGVyIGFuZCBiYWNrKS4gVGhlDQo+ Pj4+Pj4+Pj4+Pj4+Pj4+IGdvYWwgb2YgdGhlIHJlcHJvZHVjZXIgaXMgdG8gaWRlbnRpZnkgcHJv YmxlbXMgd2l0aCBzdGF0ZQ0KPj4+Pj4+Pj4+Pj4+Pj4+PiByZWNvdmVyeSB3aXRob3V0IGEgbG90 IG9mIG90aGVyIEkvTyBnb2luZyBvbiB0byBjbHV0dGVyIHVwDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHRo ZSBuZXR3b3JrIGNhcHR1cmUuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+PiBJ biB0aGUgZmFpbGluZyBjYXNlLCBzZWM9a3JiNSBpcyBzcGVjaWZpZWQgb24gdGhlIG1vdW50DQo+ Pj4+Pj4+Pj4+Pj4+Pj4+IHBvaW50LCBhbmQgdGhlIHJlcHJvZHVjZXIgaXMgcnVuIGFzIHJvb3Qu IFdlJ3ZlIGZvdW5kIHRoaXMNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gY29tYmluYXRpb24gZmFpbHMgd2l0 aCBib3RoIE5GU3Y0LjAgYW5kIE5GU3Y0LjEuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+Pj4+PiBBdCBtb3VudCB0aW1lLCB0aGUgY2xpZW50IGVzdGFibGlzaGVzIGEgR1NTIGNvbnRl eHQgZm9yDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IGxlYXNlIG1hbmFnZW1lbnQgb3BlcmF0aW9ucywgd2hp Y2ggaXMgYm91bmQgdG8gdGhlIGNsaWVudCdzDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IE5GUyBzZXJ2aWNl IHByaW5jaXBhbCBhbmQgdXNlcyBHU1Mgc2VydmljZSAiaW50ZWdyaXR5LiINCj4+Pj4+Pj4+Pj4+ Pj4+Pj4gQ2FsbCB0aGlzIEdTUyBjb250ZXh0IDEuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+ Pj4+Pj4+Pj4+PiBXaGVuIHRoZSByZXByb2R1Y2VyIHN0YXJ0cywgYSBzZWNvbmQgR1NTIGNvbnRl eHQgaXMNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gZXN0YWJsaXNoZWQgZm9yIE5GUyBvcGVyYXRpb25zIGFz c29jaWF0ZWQgd2l0aCB0aGF0IHVzZXIuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IFNpbmNlIHRoZSByZXBy b2R1Y2VyIGlzIHJ1bm5pbmcgYXMgcm9vdCwgdGhpcyBjb250ZXh0IGlzDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+IGFsc28gYm91bmQgdG8gdGhlIGNsaWVudCdzIE5GUyBzZXJ2aWNlIHByaW5jaXBhbCwgYnV0 IGl0DQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHVzZXMgdGhlIEdTUyBzZXJ2aWNlICJub25lIiAocmVmbGVj dGluZyB0aGUgZXhwbGljaXQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gcmVxdWVzdCBmb3IgInNlYz1rcmI1 IikuIENhbGwgdGhpcyBHU1MgY29udGV4dCAyLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+Pj4+Pj4gQWZ0ZXIgdGhlIHNlcnZlciByZWJvb3RzLCB0aGUgY2xpZW50IHJlLWVzdGFibGlz aGVzIGEgVENQDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IGNvbm5lY3Rpb24gd2l0aCB0aGUgc2VydmVyLCBh bmQgcGVyZm9ybXMgYSBSRU5FVw0KPj4+Pj4+Pj4+Pj4+Pj4+PiBvcGVyYXRpb24gdXNpbmcgY29u dGV4dCAxLiBUaGFua3MgdG8gdGhlIHNlcnZlciByZWJvb3QsDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IGNv bnRleHRzIDEgYW5kIDIgYXJlIG5vdyBzdGFsZS4gVGhlIHNlcnZlciB0aHVzIHJlamVjdHMNCj4+ Pj4+Pj4+Pj4+Pj4+Pj4gdGhlIFJQQyB3aXRoIFJQQ1NFQ19HU1NfQ1RYUFJPQkxFTS4NCj4+Pj4+ Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSBjbGllbnQgcGVyZm9ybXMgYSBHU1Nf SU5JVF9TRUNfQ09OVEVYVCB2aWEgYW4gTkZTdjQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gTlVMTCBvcGVy YXRpb24uIENhbGwgdGhpcyBHU1MgY29udGV4dCAzLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+ Pj4+Pj4+Pj4+Pj4gSW50ZXJlc3RpbmdseSwgdGhlIGNsaWVudCBkb2VzIG5vdCByZXNlbmQgdGhl IFJFTkVXDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IG9wZXJhdGlvbiBhdCB0aGlzIHBvaW50IChpZiBpdCBk aWQsIHdlIHdvdWxkbid0IHNlZSB0aGlzDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHByb2JsZW0gYXQgYWxs KS4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSBjbGllbnQgdGhlbiBh dHRlbXB0cyB0byByZXN1bWUgdGhlIHJlcHJvZHVjZXIgd29ya2xvYWQuDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+IEl0IHNlbmRzIGFuIE5GU3Y0IFdSSVRFIG9wZXJhdGlvbiwgdXNpbmcgdGhlIGZpcnN0IGF2 YWlsYWJsZQ0KPj4+Pj4+Pj4+Pj4+Pj4+PiBHU1MgY29udGV4dCBpbiBVSUQgMCdzIGNyZWRlbnRp YWwgY2FjaGUsIHdoaWNoIGlzIGNvbnRleHQgMywNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gYWxyZWFkeSBi b3VuZCB0byB0aGUgY2xpZW50J3MgTkZTIHNlcnZpY2UgcHJpbmNpcGFsLiBCdXQgR1NTDQo+Pj4+ Pj4+Pj4+Pj4+Pj4+IHNlcnZpY2UgIm5vbmUiIGlzIHVzZWQgZm9yIHRoaXMgb3BlcmF0aW9uLCBz aW5jZSBpdCBpcyBvbg0KPj4+Pj4+Pj4+Pj4+Pj4+PiBiZWhhbGYgb2YgdGhlIG1vdW50IHdoZXJl IHNlYz1rcmI1IHdhcyBzcGVjaWZpZWQuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+ Pj4+PiBUaGUgUlBDIGlzIGFjY2VwdGVkLCBidXQgdGhlIHNlcnZlciByZXBvcnRzDQo+Pj4+Pj4+ Pj4+Pj4+Pj4+IE5GUzRFUlJfU1RBTEVfU1RBVEVJRCwgc2luY2UgaXQgaGFzIHJlY2VudGx5IHJl Ym9vdGVkLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCBy ZXNwb25kcyBieSBhdHRlbXB0aW5nIHN0YXRlIHJlY292ZXJ5LiBUaGUNCj4+Pj4+Pj4+Pj4+Pj4+ Pj4gZmlyc3Qgb3BlcmF0aW9uIGl0IHRyaWVzIGlzIGFub3RoZXIgUkVORVcuIFNpbmNlIHRoaXMg aXMNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gYSBsZWFzZSBtYW5hZ2VtZW50IG9wZXJhdGlvbiwgdGhlIGNs aWVudCBsb29rcyBpbiBVSUQgMCdzDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IGNyZWRlbnRpYWwgY2FjaGUg YWdhaW4gYW5kIGZpbmRzIHRoZSByZWNlbnRseSBlc3RhYmxpc2hlZA0KPj4+Pj4+Pj4+Pj4+Pj4+ PiBjb250ZXh0IDMuIEl0IHRyaWVzIHRoZSBSRU5FVyBvcGVyYXRpb24gdXNpbmcgR1NTIGNvbnRl eHQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gMyB3aXRoIEdTUyBzZXJ2aWNlICJpbnRlZ3JpdHkuIg0KPj4+ Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4gVGhlIHNlcnZlciByZWplY3RzIHRoZSBS RU5FVyBSUEMgd2l0aCBBVVRIX0ZBSUxFRCwgYW5kDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHRoZSBjbGll bnQgcmVwb3J0cyB0aGF0ICJjaGVjayBsZWFzZSBmYWlsZWQiIGFuZA0KPj4+Pj4+Pj4+Pj4+Pj4+ PiB0ZXJtaW5hdGVzIHN0YXRlIHJlY292ZXJ5Lg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+Pj4+Pj4gVGhlIGNsaWVudCByZS1kcml2ZXMgdGhlIFdSSVRFIG9wZXJhdGlvbiB3aXRoIHRo ZSBzdGFsZQ0KPj4+Pj4+Pj4+Pj4+Pj4+PiBzdGF0ZWlkIHdpdGggcHJlZGljdGFibGUgcmVzdWx0 cy4gVGhlIGNsaWVudCBhZ2FpbiB0cmllcw0KPj4+Pj4+Pj4+Pj4+Pj4+PiB0byByZWNvdmVyIHN0 YXRlIGJ5IHNlbmRpbmcgYSBSRU5FVywgYW5kIHN0aWxsIHVzZXMgdGhlDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+IHNhbWUgR1NTIGNvbnRleHQgMyB3aXRoIHNlcnZpY2UgImludGVncml0eSIgYW5kIGdldHMg dGhlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHNhbWUgcmVzdWx0LiBBIChwZXJoYXBzIHNsb3ctbW90aW9u KSBTVEFMRV9TVEFURUlEIGxvb3ANCj4+Pj4+Pj4+Pj4+Pj4+Pj4gZW5zdWVzLCBhbmQgdGhlIGNs aWVudCBtb3VudCBwb2ludCBpcyBkZWFkbG9ja2VkLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+ Pj4+Pj4+Pj4+Pj4gWW91ciBhbmFseXNpcyB3YXMgdGhhdCBiZWNhdXNlIHRoZSByZXByb2R1Y2Vy IGlzIHJ1biBhcw0KPj4+Pj4+Pj4+Pj4+Pj4+PiByb290LCBib3RoIHRoZSByZXByb2R1Y2VyJ3Mg SS9PIG9wZXJhdGlvbnMsIGFuZCBsZWFzZQ0KPj4+Pj4+Pj4+Pj4+Pj4+PiBtYW5hZ2VtZW50IG9w ZXJhdGlvbnMsIGF0dGVtcHQgdG8gdXNlIHRoZSBzYW1lIEdTUyBjb250ZXh0DQo+Pj4+Pj4+Pj4+ Pj4+Pj4+IGluIFVJRCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgYnV0IGVhY2ggdXNlcyBkaWZmZXJl bnQgR1NTDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHNlcnZpY2VzLg0KPj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+ Pj4+Pj4+Pj4+Pj4+IEFzIFJGQzIyMDMgc3RhdGVzLCAiSW4gYSBjcmVhdGlvbiByZXF1ZXN0LCB0 aGUgc2VxX251bSBhbmQgc2VydmljZSBmaWVsZHMgYXJlIHVuZGVmaW5lZCBhbmQgYm90aCBtdXN0 IGJlIGlnbm9yZWQgYnkgdGhlIHNlcnZlcuKAnQ0KPj4+Pj4+Pj4+Pj4+Pj4+IFNvIGEgY29udGV4 dCBjcmVhdGlvbiByZXF1ZXN0IHdoaWxlIGtpY2tlZCBvZmYgYnkgYW4gb3BlcmF0aW9uIHdpdGgg YSBzZXJ2aWNlIGF0dGFjaGVkIChlLmcuIFdSSVRFIHVzZXMgcnBjX2dzc19zdmNfbm9uZSBhbmQg UkVORVcgdXNlcyBycGNfZ3NzX3N2Y19pbnRlZ3JpdHkpLCBjYW4gYmUgdXNlZCBieSBlaXRoZXIg c2VydmljZSBsZXZlbC4NCj4+Pj4+Pj4+Pj4+Pj4+PiBBRkFJQ1MgYSBzaW5nbGUgR1NTIGNvbnRl eHQgY291bGQgaW4gdGhlb3J5IGJlIHVzZWQgZm9yIGFsbCBzZXJ2aWNlIGxldmVscywgYnV0IGlu IHByYWN0aWNlLCBHU1MgY29udGV4dHMgYXJlIHJlc3RyaWN0ZWQgdG8gYSBzZXJ2aWNlIGxldmVs IChieSBjbGllbnQ/IGJ5IHNlcnZlcj8gKSBvbmNlIHRoZXkgYXJlIHVzZWQuDQo+Pj4+Pj4+Pj4+ Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSBrZXkgaXNzdWUg c2VlbXMgdG8gYmUgd2h5LCB3aGVuIHRoZSBtb3VudA0KPj4+Pj4+Pj4+Pj4+Pj4+PiBpcyBmaXJz dCBlc3RhYmxpc2hlZCwgdGhlIGNsaWVudCBpcyBjb3JyZWN0bHkgYWJsZSB0bw0KPj4+Pj4+Pj4+ Pj4+Pj4+PiBlc3RhYmxpc2ggdHdvIHNlcGFyYXRlIEdTUyBjb250ZXh0cyBmb3IgVUlEIDA7IGJ1 dCBhZnRlcg0KPj4+Pj4+Pj4+Pj4+Pj4+PiBhIHNlcnZlciByZWJvb3QsIHRoZSBjbGllbnQgYXR0 ZW1wdHMgdG8gdXNlIHRoZSBzYW1lIEdTUw0KPj4+Pj4+Pj4+Pj4+Pj4+PiBjb250ZXh0IHdpdGgg dHdvIGRpZmZlcmVudCBHU1Mgc2VydmljZXMuDQo+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+ Pj4+Pj4gSSBzcGVjdWxhdGUgdGhhdCBpdCBpcyBhIHJhY2UgYmV0d2VlbiB0aGUgV1JJVEUgYW5k IHRoZSBSRU5FVyB0byB1c2UgdGhlIHNhbWUgbmV3bHkgY3JlYXRlZCBHU1MgY29udGV4dCB0aGF0 IGhhcyBub3QgYmVlbiB1c2VkIHlldCwgYW5kIHNvIGhhcyBubyBhc3NpZ25lZCBzZXJ2aWNlIGxl dmVsLCBhbmQgdGhlIHR3byByZXF1ZXN0cyByYWNlIHRvIHNldCB0aGUgc2VydmljZSBsZXZlbC4N Cj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4gSSBhZ3JlZSB3aXRoIEFuZHkuIEl0IG11 c3QgYmUgYSB0aWdodCByYWNlLg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gSW4gb25l IGNhcHR1cmUgSSBzZWUgc29tZXRoaW5nIGxpa2UgdGhpcyBhZnRlcg0KPj4+Pj4+Pj4+Pj4+PiB0 aGUgc2VydmVyIHJlc3RhcnRzOg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gU1lODQo+ Pj4+Pj4+Pj4+Pj4+IFNZTiwgQUNLDQo+Pj4+Pj4+Pj4+Pj4+IEFDSw0KPj4+Pj4+Pj4+Pj4+PiBD IFdSSVRFDQo+Pj4+Pj4+Pj4+Pj4+IEMgU0VRVUVOQ0UNCj4+Pj4+Pj4+Pj4+Pj4gUiBXUklURSAt PiBDVFhfUFJPQkxFTQ0KPj4+Pj4+Pj4+Pj4+PiBSIFNFUVVFTkNFIC0+IENUWF9QUk9CTEVNDQo+ Pj4+Pj4+Pj4+Pj4+IEMgTlVMTCAoS1JCNV9BUF9SRVEpDQo+Pj4+Pj4+Pj4+Pj4+IFIgTlVMTCAo S1JCNV9BUF9SRVApDQo+Pj4+Pj4+Pj4+Pj4+IEMgV1JJVEUNCj4+Pj4+Pj4+Pj4+Pj4gQyBTRVFV RU5DRQ0KPj4+Pj4+Pj4+Pj4+PiBSIFdSSVRFIC0+IE5GUzRFUlJfU1RBTEVfU1RBVEVJRA0KPj4+ Pj4+Pj4+Pj4+PiBSIFNFUVVFTkNFIC0+IEFVVEhfRkFJTEVEDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+ Pj4+Pj4+Pj4+PiBBbmR5J3MgdGhlb3J5IG5lYXRseSBleHBsYWlucyB0aGlzIGJlaGF2aW9yLg0K Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+PiBJIGhhdmUgdHJp ZWQgdG8gcmVwcm9kdWNlDQo+Pj4+Pj4+Pj4+Pj4+PiB5b3VyIHNjZW5hcmlvIGFuZCBpbiBteSB0 ZXN0cyBvZiByZWJvb3RpbmcgdGhlIHNlcnZlciBhbGwgcmVjb3Zlcg0KPj4+Pj4+Pj4+Pj4+Pj4g Y29ycmVjdGx5LiBJbiBteSBjYXNlLCBpZiBSRU5FVyB3YXMgdGhlIG9uZSBoaXR0aW5nIHRoZSBB VVRIX0VSUiB0aGVuDQo+Pj4+Pj4+Pj4+Pj4+PiB0aGUgbmV3IGNvbnRleHQgaXMgZXN0YWJsaXNo ZWQgYW5kIHRoZW4gUkVORVcgdXNpbmcgaW50ZWdyaXR5IHNlcnZpY2UNCj4+Pj4+Pj4+Pj4+Pj4+ IGlzIHJldHJpZWQgd2l0aCB0aGUgbmV3IGNvbnRleHQgd2hpY2ggZ2V0cyBFUlJfU1RBTEVfQ0xJ RU5USUQgd2hpY2gNCj4+Pj4+Pj4+Pj4+Pj4+IHRoZW4gY2xpZW50IHJlY292ZXJzIGZyb20uIElm IGl0J3MgYW4gb3BlcmF0aW9uIChJIGhhdmUgYSBHRVRBVFRSKQ0KPj4+Pj4+Pj4+Pj4+Pj4gdGhh dCBnZXRzIEFVVEhfRVJSLCB0aGVuIGl0IGdldHMgbmV3IGNvbnRleHQgYW5kIGlzIHJldHJpZWQg dXNpbmcgbm9uZQ0KPj4+Pj4+Pj4+Pj4+Pj4gc2VydmljZS4gVGhlbiBSRU5FVyBnZXRzIGl0cyBv d24gQVVUSF9FUlIgYXMgaXQgdXNlcyBhIGRpZmZlcmVudA0KPj4+Pj4+Pj4+Pj4+Pj4gY29udGV4 dCwgYSBuZXcgY29udGV4dCBpcyBnb3R0ZW4sIFJFTkVXIGlzIHJldHJpZWQgb3ZlciBpbnRlZ3Jp dHkgYW5kDQo+Pj4+Pj4+Pj4+Pj4+PiBnZXRzIEVSUl9TVEFMRV9DTElFTlRJRCB3aGljaCBpdCBy ZWNvdmVycyBmcm9tLg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4gSWYgb25lIG9wZXJh dGlvbiBpcyBhbGxvd2VkIHRvIGNvbXBsZXRlLCB0aGVuDQo+Pj4+Pj4+Pj4+Pj4+IHRoZSBvdGhl ciB3aWxsIGFsd2F5cyByZWNvZ25pemUgdGhhdCBhbm90aGVyDQo+Pj4+Pj4+Pj4+Pj4+IGZyZXNo IEdTUyBjb250ZXh0IGlzIG5lZWRlZC4gSWYgdHdvIGFyZSBzZW50DQo+Pj4+Pj4+Pj4+Pj4+IGF0 IHRoZSBzYW1lIHRpbWUsIHRoZXkgcmFjZSBhbmQgb25lIGFsd2F5cw0KPj4+Pj4+Pj4+Pj4+PiBm YWlscy4NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IEhlbGVuJ3MgdGVzdCBpbmNsdWRl cyBhIHNlY29uZCBpZGxlIG1vdW50IHBvaW50DQo+Pj4+Pj4+Pj4+Pj4+IChzZWM9a3JiNWkpIGFu ZCBtYXliZSB0aGF0IGlzIG5lZWRlZCB0byB0cmlnZ2VyDQo+Pj4+Pj4+Pj4+Pj4+IHRoZSByYWNl Pw0KPj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+IENodWNrLCBhbnkgY2hhbmNlIHRvIGdldCAi cnBjZGVidWcgLW0gcnBjIGF1dGgiIG91dHB1dCBkdXJpbmcgdGhlDQo+Pj4+Pj4+Pj4+Pj4gZmFp bHVyZSAoZ3NzZCBvcHRpb25hbGx5KSAoaSByZWFsaXplIHRoYXQgaXQgbWlnaHQgYWx0ZXIgdGhl IHRpbWluZ3MNCj4+Pj4+Pj4+Pj4+PiBhbmQgbm90IGhpdCB0aGUgaXNzdWUgYnV0IHdvcnRoIGEg c2hvdCk/DQo+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+IEknbSBzdXJlIHRoYXQncyBmaW5lLiBB biBpbnRlcm5hbCB0ZXN0ZXIgaGl0IHRoaXMsDQo+Pj4+Pj4+Pj4+PiBub3QgYSBjdXN0b21lciwg c28gSSB3aWxsIGFzay4NCj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4gSSBhZ3JlZSwgdGhvdWdo LCB0aGF0IHRpbWluZyBtaWdodCBiZSBhIHByb2JsZW06DQo+Pj4+Pj4+Pj4+PiB0aGVzZSBzeXN0 ZW1zIGFsbCBoYXZlIHJlYWwgc2VyaWFsIGNvbnNvbGVzIHZpYQ0KPj4+Pj4+Pj4+Pj4gaUxPTSwg c28gL3YvbC9tIHRyYWZmaWMgZG9lcyBicmluZyBldmVyeXRoaW5nIHRvDQo+Pj4+Pj4+Pj4+PiBh IHN0YW5kc3RpbGwuDQo+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+IE1lYW53aGlsZSwgd2hhdCdz IHlvdSdyZSBvcGluaW9uIGFib3V0IEFVVEhfRkFJTEVEPw0KPj4+Pj4+Pj4+Pj4gU2hvdWxkIHRo ZSBzZXJ2ZXIgcmV0dXJuIFJQQ1NFQ19HU1NfQ1RYUFJPQkxFTQ0KPj4+Pj4+Pj4+Pj4gaW4gdGhp cyBjYXNlIGluc3RlYWQ/IElmIGl0IGRpZCwgZG8geW91IHRoaW5rDQo+Pj4+Pj4+Pj4+PiB0aGUg TGludXggY2xpZW50IHdvdWxkIHJlY292ZXIgYnkgY3JlYXRpbmcgYQ0KPj4+Pj4+Pj4+Pj4gcmVw bGFjZW1lbnQgR1NTIGNvbnRleHQ/DQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+PiBBaCwgeWVzLCBJ IGVxdWF0ZWQgQVVUSF9GQUlMRUQgQW5kIEFVVEhfRVJST1IgaW4gbXkgbWluZC4gSWYgY2xpZW50 DQo+Pj4+Pj4+Pj4+IHJlY2VpdmVzIHRoZSByZWFzb24gYXMgQVVUSF9GQUlMRUQgYXMgb3Bwb3Nl IHRvIENUWFBST0JMRU0gaXQgd2lsbA0KPj4+Pj4+Pj4+PiBmYWlsIHdpdGggRUlPIGVycm9yIGFu ZCB3aWxsIG5vdCB0cnkgdG8gY3JlYXRlIGEgbmV3IEdTUyBjb250ZXh0LiBTbw0KPj4+Pj4+Pj4+ PiB5ZXMsIEkgYmVsaWV2ZSBpdCB3b3VsZCBoZWxwIGlmIHRoZSBzZXJ2ZXIgcmV0dXJucyBhbnkg b2YgdGhlDQo+Pj4+Pj4+Pj4+IGZvbGxvd2luZyBlcnJvcnM6DQo+Pj4+Pj4+Pj4+ICAgICAgICAg ICAgIGNhc2UgUlBDX0FVVEhfUkVKRUNURURDUkVEOg0KPj4+Pj4+Pj4+PiAgICAgICAgICAgICBj YXNlIFJQQ19BVVRIX1JFSkVDVEVEVkVSRjoNCj4+Pj4+Pj4+Pj4gICAgICAgICAgICAgY2FzZSBS UENTRUNfR1NTX0NSRURQUk9CTEVNOg0KPj4+Pj4+Pj4+PiAgICAgICAgICAgICBjYXNlIFJQQ1NF Q19HU1NfQ1RYUFJPQkxFTToNCj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+IHRoZW4gdGhlIGNsaWVu dCB3aWxsIHJlY3JlYXRlIHRoZSBjb250ZXh0Lg0KPj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+IEFsc28g aW4gbXkgdGVzdGluZywgSSBjYW4gc2VlIHRoYXQgY3JlZGVudGlhbCBjYWNoZSBpcyBwZXIgZ3Nz IGZsYXZvci4NCj4+Pj4+Pj4+PiBKdXN0IHRvIGNoZWNrLCB3aGF0IGtlcm5lbCB2ZXJzaW9uIGlz IHRoaXMgcHJvYmxlbSBlbmNvdW50ZXJlZCBvbiAoSQ0KPj4+Pj4+Pj4+IGtub3cgeW91IHNhaWQg dXBzdHJlYW0pIGJ1dCBJIGp1c3Qgd2FudCB0byBkb3VibGUgY2hlY2sgc28gdGhhdCBJIGNhbg0K Pj4+Pj4+Pj4+IGxvb2sgYXQgdGhlIGNvcnJlY3Qgc291cmNlIGNvZGUuDQo+Pj4+Pj4+PiANCj4+ Pj4+Pj4+IHY0LjEuMTIgKHN0YWJsZSkgSSB0aGluay4NCj4+Pj4+Pj4gDQo+Pj4+Pj4+IEFsc28s IGNhbiB5b3Ugc2hhcmUgdGhlIG5ldHdvcmsgdHJhY2U/DQo+Pj4+Pj4gDQo+Pj4+Pj4gSGkgQ2h1 Y2ssDQo+Pj4+Pj4gDQo+Pj4+Pj4gSSB3YXMgZmluYWxseSBhYmxlIHRvIHJlcHJvZHVjZSB0aGUg Y29uZGl0aW9uIHlvdSB3ZXJlIHNlZWluZyAoaS5lLiwNCj4+Pj4+PiB0aGUgdXNlIG9mIHRoZSBz YW1lIGNvbnRleHQgZm9yIGRpZmZlcmVudCBnc3Mgc2VydmljZXMpLg0KPj4+Pj4+IA0KPj4+Pj4+ IEkgZW5hYmxlZCBycGNkZWJ1ZyBycGMgYXV0aCBhbmQgSSBjYW4gc2VlIHRoYXQgdGhlIDJuZCBy ZXF1ZXN0IGVuZHMgdXANCj4+Pj4+PiBmaW5kaW5nIGEgZ3NzX3VwY2FsbCBtZXNzYWdlIGJlY2F1 c2UgaXQncyBqdXN0IG1hdGNoZWQgYnkgdGhlIHVpZC4NCj4+Pj4+PiBUaGVyZSBpcyBldmVuIGEg Y29tbWVudCBpbiBhdXRoX2dzcy9hdXRoX2dzcy5jIGluIGdzc19hZGRfbXNnKCkgc2F5aW5nDQo+ Pj4+Pj4gdGhhdCBpZiB0aGVyZSBpcyB1cGNhbGwgZm9yIGFuIHVpZCB0aGVuIGl0IHdvbid0IGFk ZCBhbm90aGVyIHVwY2FsbC4NCj4+Pj4+PiBTbyBJIHRoaW5rIHRoZSBkZWNpc2lvbiBpcyBtYWRl IHJpZ2h0IHRoZXJlIHRvIHNoYXJlIHRoZSBzYW1lIGNvbnRleHQNCj4+Pj4+PiBubyBtYXR0ZXIg dGhlIGdzcyBzZXJ2aWNlLg0KPj4+Pj4gDQo+Pj4+PiBJZiBJIHVuZGVyc3RhbmQgY29ycmVjdGx5 LCB0aGF0J3MganVzdCB3aGF0IEFuZHkgcHJlZGljdGVkLg0KPj4+Pj4gDQo+Pj4+PiBUaGF0IGNo ZWNrIG5lZWRzIHRvIGJlIGNoYW5nZWQgdG8gYWxsb3cgYW5vdGhlciB1cGNhbGwgdG8gYmUNCj4+ Pj4+IHF1ZXVlZCBpZiB0aGUgVUlEIG1hdGNoZXMgYnV0IHRoZSBHU1Mgc2VydmljZSBkb2VzIG5v dC4NCj4+Pj4gDQo+Pj4+IFlvdSBzaG91bGQgYmUgYWJsZSB0byB1c2UgdGhlIHNhbWUgY29udGV4 dCB3aXRoIGRpZmZlcmVudCBzZXJ2aWNlcy4NCj4+Pj4gDQo+Pj4+IEFwb2xvZ2llcywgSSBoYXZl bid0IGNhdWdodCB1cCB3aXRoIHRoZSB3aG9sZSBkaXNjdXNzaW9uIGFib3ZlLCB0aGlzIG9uZQ0K Pj4+PiBwb2ludCBqdXN0IGp1bXBlZCBvdXQgYXQgbWUuICBJZiB5b3UncmUgdHJ5aW5nIHRvIHJl cXVlc3QgYSB3aG9sZSBuZXcNCj4+Pj4gZ3NzIGNvbnRleHQganVzdCBzbyB5b3UgY2FuIHVzZSwg ZS5nLiwgaW50ZWdyaXR5IGluc3RlYWQgb2YgcHJpdmFjeSwNCj4+Pj4gdGhlbiBzb21ldGhpbmcn cyB3cm9uZy4NCj4+PiANCj4+PiBUaGUgY2xpZW50IGNvZGUgaGFzIHNlcGFyYXRlIGdzc19jcmVk IGNhY2hlcyAoYW5kIHNvIHNlcGFyYXRlIGdzc19jb250ZXjigJlzKSBwZXIgZ3NzX2F1dGgsIHdo aWNoIGlzIHBlciBzZXJ2aWNlLiBBRkFJSyB0aGUgY2xpZW50IGhhcyBhbHdheXMgb2J0YWluZWQg YSBzZXBhcmF0ZSBjb250ZXh0IHBlciBzZXJ2aWNlIHBlciBzZXJ2ZXIuIFdoaWxlIHdlIGNhbiB1 c2UgdGhlIHNhbWUgZ3NzIGNvbnRleHQgd2l0aCBkaWZmZXJlbnQgc2VydmljZXMsIHRoYXQgaXMg bm90IHRoZSBkZXNpZ24gY2hvaWNlLg0KPj4gDQo+PiBHaXZlbiB0aGUgY3VycmVudCBjb2RlLCBJ J2Qgc2F5IHRoYXQgaXQncyBub3QgY2xlYXIgd2hhdCB0aGUgZGVzaWduDQo+PiBjaG9pY2UgaXMu IFRoZSB1cGNhbGwgY29kZSBzdGF0ZXMgdGhhdCBpdCB3aWxsIG5vdCBkbyBhbm90aGVyIHVwY2Fs bA0KPj4gZm9yIGEgZ2l2ZW4gVUlEIGlmIGFub3RoZXIgdXBjYWxsIGlzIGFscmVhZHkgbWFkZS4g U28gaXQncyBwdXJlbHkNCj4+IHRpbWluZyBsdWNrIHRoYXQgd2UgaGF2ZSB0d28gZGlmZmVyZW50 IGNvbnRleHRzIGZvciB0aGUgbW91bnQgd2hlbg0KPj4gc2VjPWtyYjUgYW5kIHRoZSBkZWZhdWx0 IGZvciBsZWFzZSBvcGVyYXRpb25zIGlzICJpbnRlZ3JpdHkiLiBBbmQgd2UNCj4+IGRvbid0IHBh c3MgdGhlIEdTUyBzZXJ2aWNlIGJldHdlZW4gdGhlIGtlcm5lbCBhbmQgZ3NzZCBhbmQgdGh1cyBu byB3YXkNCj4+IGZvciB1cyB0byB0aWUgdGhlIHVwY2FsbCB0byB0aGUgc2VydmljZS4NCj4gDQo+ IFdlbGwsIEkgd3JvdGUgc29tZSBvZiB0aGUgdXBjYWxsIGNvZGUsIGFuZCBJIHRob3VnaCAoaW5j b3JyZWN0bHksIEkNCj4gZ3Vlc3MpIHRoYXQgY2xpZW50cyBjb3VsZCByZXVzZSB0aGUgc2FtZSBj b250ZXh0IHdpdGggbXVsdGlwbGUgc2VydmljZXMsDQo+IHNvIHRoZSBjb25mdXNpb24gbWF5IGp1 c3QgYmUgbXkgZmF1bHQuICBTbyB5b3UgdGhpbmsgdGhlIG9ubHkgZml4IGhlcmUNCj4gaXMgdG8g a2V5IHRoZSB1cGNhbGxzDQoNCk5vIG5lZWQgdG8gY2hhbmdlIHRoZSB1cGNhbGwsIGp1c3QgZG9u 4oCZIHQgbGV0IHRoZSAybmQgcmVxdWVzdCB1c2UgdGhlIG5ldyBkb3duY2FsbCBnc3MgY29udGV4 dCB1bmxlc3MgdGhlIHNlcnZpY2UgYXNzaWduZWQgdG8gaXQgbWF0Y2hlcy4gSSB3b3VsZCB0aGlu ayB5b3UgY291bGQgc2V0IHRoZSBzZXJ2aWNlIGltbWVkaWF0ZWx5IHVwb24gcmVjZWl2aW5nIHRo ZSBuZXcgZ3NzIGNvbnRleHQgZG93biBjYWxsIGFuZCB0aGVuIGNoZWNrIGl0IHByaW9yIHRvIHRo ZSAybmQgcmVxdWVzdCB1c2VzIGl0Lg0KDQrigJQ+QW5keQ0KDQoNCg0KPiBvbiAobWVjaGFuaXNt LCBzZXJ2aWNlLCB1aWQpIGluc3RlYWQgb2YganVzdA0KPiAobWVjaGFuaXNtLCB1aWQpPw0KPiAN Cj4gLS1iLg0KDQo= ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 20:18 ` Adamson, Andy @ 2016-08-03 20:33 ` Trond Myklebust 2016-08-03 21:12 ` Adamson, Andy 0 siblings, 1 reply; 25+ messages in thread From: Trond Myklebust @ 2016-08-03 20:33 UTC (permalink / raw) To: Adamson William Andros Cc: Fields Bruce James, Kornievskaia Olga, Lever Chuck, List Linux NFS Mailing DQo+IE9uIEF1ZyAzLCAyMDE2LCBhdCAxNjoxOCwgQWRhbXNvbiwgQW5keSA8V2lsbGlhbS5BZGFt c29uQG5ldGFwcC5jb20+IHdyb3RlOg0KPiANCj4gDQo+PiBPbiBBdWcgMywgMjAxNiwgYXQgNDow NiBQTSwgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRzQGZpZWxkc2VzLm9yZz4gd3JvdGU6DQo+PiAN Cj4+IE9uIFdlZCwgQXVnIDAzLCAyMDE2IGF0IDAzOjU2OjMxUE0gLTA0MDAsIE9sZ2EgS29ybmll dnNrYWlhIHdyb3RlOg0KPj4+IE9uIFdlZCwgQXVnIDMsIDIwMTYgYXQgMjo1MyBQTSwgQWRhbXNv biwgQW5keQ0KPj4+IDxXaWxsaWFtLkFkYW1zb25AbmV0YXBwLmNvbT4gd3JvdGU6DQo+Pj4+IA0K Pj4+Pj4gT24gQXVnIDIsIDIwMTYsIGF0IDI6MDYgUE0sIEouIEJydWNlIEZpZWxkcyA8YmZpZWxk c0BmaWVsZHNlcy5vcmc+IHdyb3RlOg0KPj4+Pj4gDQo+Pj4+PiBPbiBGcmksIEp1bCAyOSwgMjAx NiBhdCAxMjozODozNFBNIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4+Pj4+PiANCj4+Pj4+ Pj4gT24gSnVsIDI5LCAyMDE2LCBhdCAxMjoyNyBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9A dW1pY2guZWR1PiB3cm90ZToNCj4+Pj4+Pj4gDQo+Pj4+Pj4+IE9uIE1vbiwgSnVsIDI1LCAyMDE2 IGF0IDI6MTggUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+ Pj4+Pj4+PiBPbiBUaHUsIEp1bCAyMSwgMjAxNiBhdCA1OjMyIFBNLCBDaHVjayBMZXZlciA8Y2h1 Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4gDQo+Pj4+ Pj4+Pj4+IE9uIEp1bCAyMSwgMjAxNiwgYXQgMTA6NDYgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxh Z2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4gT24gVGh1LCBK dWwgMjEsIDIwMTYgYXQgMzo1NCBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2guZWR1 PiB3cm90ZToNCj4+Pj4+Pj4+Pj4+PiBPbiBUaHUsIEp1bCAyMSwgMjAxNiBhdCAxOjU2IFBNLCBD aHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4g DQo+Pj4+Pj4+Pj4+Pj4+IE9uIEp1bCAyMSwgMjAxNiwgYXQgNjowNCBQTSwgT2xnYSBLb3JuaWV2 c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+ Pj4+IE9uIFRodSwgSnVsIDIxLCAyMDE2IGF0IDI6NTUgQU0sIENodWNrIExldmVyIDxjaHVjay5s ZXZlckBvcmFjbGUuY29tPiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+ IE9uIEp1bCAyMCwgMjAxNiwgYXQgNjo1NiBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1p Y2guZWR1PiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+PiBPbiBXZWQs IEp1bCAyMCwgMjAxNiBhdCA1OjE0IEFNLCBBZGFtc29uLCBBbmR5DQo+Pj4+Pj4+Pj4+Pj4+Pj4g PFdpbGxpYW0uQWRhbXNvbkBuZXRhcHAuY29tPiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+ Pj4+Pj4+Pj4+Pj4+Pj4+PiBPbiBKdWwgMTksIDIwMTYsIGF0IDEwOjUxIEFNLCBDaHVjayBMZXZl ciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+ Pj4+Pj4+Pj4+Pj4+Pj4+IEhpIEFuZHktDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ Pj4+Pj4+IFRoYW5rcyBmb3IgdGFraW5nIHRoZSB0aW1lIHRvIGRpc2N1c3MgdGhpcyB3aXRoIG1l LiBJJ3ZlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBjb3BpZWQgbGludXgtbmZzIHRvIG1ha2UgdGhpcyBl LW1haWwgYWxzbyBhbiB1cHN0cmVhbSBidWcNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHJlcG9ydC4NCj4+ Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gQXMgd2Ugc2F3IGluIHRoZSBuZXR3 b3JrIGNhcHR1cmUsIHJlY292ZXJ5IG9mIEdTUyBjb250ZXh0cw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4g YWZ0ZXIgYSBzZXJ2ZXIgcmVib290IGZhaWxzIGluIGNlcnRhaW4gY2FzZXMgd2l0aCBORlN2NC4w DQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBhbmQgTkZTdjQuMSBtb3VudCBwb2ludHMuDQo+Pj4+Pj4+Pj4+ Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSByZXByb2R1Y2VyIGlzIGEgc2ltcGxlIHBy b2dyYW0gdGhhdCBnZW5lcmF0ZXMgb25lIE5GUw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gV1JJVEUgcGVy aW9kaWNhbGx5LCBydW4gd2hpbGUgdGhlIHNlcnZlciByZXBlYXRlZGx5IHJlYm9vdHMNCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+IChvciBvbmUgY2x1c3RlciBoZWFkIGZhaWxzIG92ZXIgdG8gdGhlIG90aGVy IGFuZCBiYWNrKS4gVGhlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBnb2FsIG9mIHRoZSByZXByb2R1Y2Vy IGlzIHRvIGlkZW50aWZ5IHByb2JsZW1zIHdpdGggc3RhdGUNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHJl Y292ZXJ5IHdpdGhvdXQgYSBsb3Qgb2Ygb3RoZXIgSS9PIGdvaW5nIG9uIHRvIGNsdXR0ZXIgdXAN Cj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHRoZSBuZXR3b3JrIGNhcHR1cmUuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+ PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IEluIHRoZSBmYWlsaW5nIGNhc2UsIHNlYz1rcmI1IGlzIHNw ZWNpZmllZCBvbiB0aGUgbW91bnQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHBvaW50LCBhbmQgdGhlIHJl cHJvZHVjZXIgaXMgcnVuIGFzIHJvb3QuIFdlJ3ZlIGZvdW5kIHRoaXMNCj4+Pj4+Pj4+Pj4+Pj4+ Pj4+IGNvbWJpbmF0aW9uIGZhaWxzIHdpdGggYm90aCBORlN2NC4wIGFuZCBORlN2NC4xLg0KPj4+ Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBBdCBtb3VudCB0aW1lLCB0aGUgY2xp ZW50IGVzdGFibGlzaGVzIGEgR1NTIGNvbnRleHQgZm9yDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBsZWFz ZSBtYW5hZ2VtZW50IG9wZXJhdGlvbnMsIHdoaWNoIGlzIGJvdW5kIHRvIHRoZSBjbGllbnQncw0K Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gTkZTIHNlcnZpY2UgcHJpbmNpcGFsIGFuZCB1c2VzIEdTUyBzZXJ2 aWNlICJpbnRlZ3JpdHkuIg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gQ2FsbCB0aGlzIEdTUyBjb250ZXh0 IDEuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IFdoZW4gdGhlIHJlcHJv ZHVjZXIgc3RhcnRzLCBhIHNlY29uZCBHU1MgY29udGV4dCBpcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4g ZXN0YWJsaXNoZWQgZm9yIE5GUyBvcGVyYXRpb25zIGFzc29jaWF0ZWQgd2l0aCB0aGF0IHVzZXIu DQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBTaW5jZSB0aGUgcmVwcm9kdWNlciBpcyBydW5uaW5nIGFzIHJv b3QsIHRoaXMgY29udGV4dCBpcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gYWxzbyBib3VuZCB0byB0aGUg Y2xpZW50J3MgTkZTIHNlcnZpY2UgcHJpbmNpcGFsLCBidXQgaXQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+ IHVzZXMgdGhlIEdTUyBzZXJ2aWNlICJub25lIiAocmVmbGVjdGluZyB0aGUgZXhwbGljaXQNCj4+ Pj4+Pj4+Pj4+Pj4+Pj4+IHJlcXVlc3QgZm9yICJzZWM9a3JiNSIpLiBDYWxsIHRoaXMgR1NTIGNv bnRleHQgMi4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gQWZ0ZXIgdGhl IHNlcnZlciByZWJvb3RzLCB0aGUgY2xpZW50IHJlLWVzdGFibGlzaGVzIGEgVENQDQo+Pj4+Pj4+ Pj4+Pj4+Pj4+PiBjb25uZWN0aW9uIHdpdGggdGhlIHNlcnZlciwgYW5kIHBlcmZvcm1zIGEgUkVO RVcNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IG9wZXJhdGlvbiB1c2luZyBjb250ZXh0IDEuIFRoYW5rcyB0 byB0aGUgc2VydmVyIHJlYm9vdCwNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGNvbnRleHRzIDEgYW5kIDIg YXJlIG5vdyBzdGFsZS4gVGhlIHNlcnZlciB0aHVzIHJlamVjdHMNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+ IHRoZSBSUEMgd2l0aCBSUENTRUNfR1NTX0NUWFBST0JMRU0uDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSBjbGllbnQgcGVyZm9ybXMgYSBHU1NfSU5JVF9TRUNfQ09O VEVYVCB2aWEgYW4gTkZTdjQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IE5VTEwgb3BlcmF0aW9uLiBDYWxs IHRoaXMgR1NTIGNvbnRleHQgMy4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+ Pj4gSW50ZXJlc3RpbmdseSwgdGhlIGNsaWVudCBkb2VzIG5vdCByZXNlbmQgdGhlIFJFTkVXDQo+ Pj4+Pj4+Pj4+Pj4+Pj4+PiBvcGVyYXRpb24gYXQgdGhpcyBwb2ludCAoaWYgaXQgZGlkLCB3ZSB3 b3VsZG4ndCBzZWUgdGhpcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gcHJvYmxlbSBhdCBhbGwpLg0KPj4+ Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBUaGUgY2xpZW50IHRoZW4gYXR0ZW1w dHMgdG8gcmVzdW1lIHRoZSByZXByb2R1Y2VyIHdvcmtsb2FkLg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4g SXQgc2VuZHMgYW4gTkZTdjQgV1JJVEUgb3BlcmF0aW9uLCB1c2luZyB0aGUgZmlyc3QgYXZhaWxh YmxlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBHU1MgY29udGV4dCBpbiBVSUQgMCdzIGNyZWRlbnRpYWwg Y2FjaGUsIHdoaWNoIGlzIGNvbnRleHQgMywNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGFscmVhZHkgYm91 bmQgdG8gdGhlIGNsaWVudCdzIE5GUyBzZXJ2aWNlIHByaW5jaXBhbC4gQnV0IEdTUw0KPj4+Pj4+ Pj4+Pj4+Pj4+Pj4gc2VydmljZSAibm9uZSIgaXMgdXNlZCBmb3IgdGhpcyBvcGVyYXRpb24sIHNp bmNlIGl0IGlzIG9uDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBiZWhhbGYgb2YgdGhlIG1vdW50IHdoZXJl IHNlYz1rcmI1IHdhcyBzcGVjaWZpZWQuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ Pj4+Pj4+IFRoZSBSUEMgaXMgYWNjZXB0ZWQsIGJ1dCB0aGUgc2VydmVyIHJlcG9ydHMNCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+IE5GUzRFUlJfU1RBTEVfU1RBVEVJRCwgc2luY2UgaXQgaGFzIHJlY2VudGx5 IHJlYm9vdGVkLg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBUaGUgY2xp ZW50IHJlc3BvbmRzIGJ5IGF0dGVtcHRpbmcgc3RhdGUgcmVjb3ZlcnkuIFRoZQ0KPj4+Pj4+Pj4+ Pj4+Pj4+Pj4gZmlyc3Qgb3BlcmF0aW9uIGl0IHRyaWVzIGlzIGFub3RoZXIgUkVORVcuIFNpbmNl IHRoaXMgaXMNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGEgbGVhc2UgbWFuYWdlbWVudCBvcGVyYXRpb24s IHRoZSBjbGllbnQgbG9va3MgaW4gVUlEIDAncw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gY3JlZGVudGlh bCBjYWNoZSBhZ2FpbiBhbmQgZmluZHMgdGhlIHJlY2VudGx5IGVzdGFibGlzaGVkDQo+Pj4+Pj4+ Pj4+Pj4+Pj4+PiBjb250ZXh0IDMuIEl0IHRyaWVzIHRoZSBSRU5FVyBvcGVyYXRpb24gdXNpbmcg R1NTIGNvbnRleHQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IDMgd2l0aCBHU1Mgc2VydmljZSAiaW50ZWdy aXR5LiINCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gVGhlIHNlcnZlciBy ZWplY3RzIHRoZSBSRU5FVyBSUEMgd2l0aCBBVVRIX0ZBSUxFRCwgYW5kDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+PiB0aGUgY2xpZW50IHJlcG9ydHMgdGhhdCAiY2hlY2sgbGVhc2UgZmFpbGVkIiBhbmQNCj4+ Pj4+Pj4+Pj4+Pj4+Pj4+IHRlcm1pbmF0ZXMgc3RhdGUgcmVjb3ZlcnkuDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSBjbGllbnQgcmUtZHJpdmVzIHRoZSBXUklURSBv cGVyYXRpb24gd2l0aCB0aGUgc3RhbGUNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHN0YXRlaWQgd2l0aCBw cmVkaWN0YWJsZSByZXN1bHRzLiBUaGUgY2xpZW50IGFnYWluIHRyaWVzDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+PiB0byByZWNvdmVyIHN0YXRlIGJ5IHNlbmRpbmcgYSBSRU5FVywgYW5kIHN0aWxsIHVzZXMg dGhlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBzYW1lIEdTUyBjb250ZXh0IDMgd2l0aCBzZXJ2aWNlICJp bnRlZ3JpdHkiIGFuZCBnZXRzIHRoZQ0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gc2FtZSByZXN1bHQuIEEg KHBlcmhhcHMgc2xvdy1tb3Rpb24pIFNUQUxFX1NUQVRFSUQgbG9vcA0KPj4+Pj4+Pj4+Pj4+Pj4+ Pj4gZW5zdWVzLCBhbmQgdGhlIGNsaWVudCBtb3VudCBwb2ludCBpcyBkZWFkbG9ja2VkLg0KPj4+ Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBZb3VyIGFuYWx5c2lzIHdhcyB0aGF0 IGJlY2F1c2UgdGhlIHJlcHJvZHVjZXIgaXMgcnVuIGFzDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiByb290 LCBib3RoIHRoZSByZXByb2R1Y2VyJ3MgSS9PIG9wZXJhdGlvbnMsIGFuZCBsZWFzZQ0KPj4+Pj4+ Pj4+Pj4+Pj4+Pj4gbWFuYWdlbWVudCBvcGVyYXRpb25zLCBhdHRlbXB0IHRvIHVzZSB0aGUgc2Ft ZSBHU1MgY29udGV4dA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gaW4gVUlEIDAncyBjcmVkZW50aWFsIGNh Y2hlLCBidXQgZWFjaCB1c2VzIGRpZmZlcmVudCBHU1MNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHNlcnZp Y2VzLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4gQXMgUkZDMjIwMyBzdGF0 ZXMsICJJbiBhIGNyZWF0aW9uIHJlcXVlc3QsIHRoZSBzZXFfbnVtIGFuZCBzZXJ2aWNlIGZpZWxk cyBhcmUgdW5kZWZpbmVkIGFuZCBib3RoIG11c3QgYmUgaWdub3JlZCBieSB0aGUgc2VydmVy4oCd DQo+Pj4+Pj4+Pj4+Pj4+Pj4+IFNvIGEgY29udGV4dCBjcmVhdGlvbiByZXF1ZXN0IHdoaWxlIGtp Y2tlZCBvZmYgYnkgYW4gb3BlcmF0aW9uIHdpdGggYSBzZXJ2aWNlIGF0dGFjaGVkIChlLmcuIFdS SVRFIHVzZXMgcnBjX2dzc19zdmNfbm9uZSBhbmQgUkVORVcgdXNlcyBycGNfZ3NzX3N2Y19pbnRl Z3JpdHkpLCBjYW4gYmUgdXNlZCBieSBlaXRoZXIgc2VydmljZSBsZXZlbC4NCj4+Pj4+Pj4+Pj4+ Pj4+Pj4gQUZBSUNTIGEgc2luZ2xlIEdTUyBjb250ZXh0IGNvdWxkIGluIHRoZW9yeSBiZSB1c2Vk IGZvciBhbGwgc2VydmljZSBsZXZlbHMsIGJ1dCBpbiBwcmFjdGljZSwgR1NTIGNvbnRleHRzIGFy ZSByZXN0cmljdGVkIHRvIGEgc2VydmljZSBsZXZlbCAoYnkgY2xpZW50PyBieSBzZXJ2ZXI/ICkg b25jZSB0aGV5IGFyZSB1c2VkLg0KPj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4g DQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBUaGUga2V5IGlzc3VlIHNlZW1zIHRvIGJlIHdoeSwgd2hlbiB0 aGUgbW91bnQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGlzIGZpcnN0IGVzdGFibGlzaGVkLCB0aGUgY2xp ZW50IGlzIGNvcnJlY3RseSBhYmxlIHRvDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiBlc3RhYmxpc2ggdHdv IHNlcGFyYXRlIEdTUyBjb250ZXh0cyBmb3IgVUlEIDA7IGJ1dCBhZnRlcg0KPj4+Pj4+Pj4+Pj4+ Pj4+Pj4gYSBzZXJ2ZXIgcmVib290LCB0aGUgY2xpZW50IGF0dGVtcHRzIHRvIHVzZSB0aGUgc2Ft ZSBHU1MNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGNvbnRleHQgd2l0aCB0d28gZGlmZmVyZW50IEdTUyBz ZXJ2aWNlcy4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IEkgc3BlY3VsYXRl IHRoYXQgaXQgaXMgYSByYWNlIGJldHdlZW4gdGhlIFdSSVRFIGFuZCB0aGUgUkVORVcgdG8gdXNl IHRoZSBzYW1lIG5ld2x5IGNyZWF0ZWQgR1NTIGNvbnRleHQgdGhhdCBoYXMgbm90IGJlZW4gdXNl ZCB5ZXQsIGFuZCBzbyBoYXMgbm8gYXNzaWduZWQgc2VydmljZSBsZXZlbCwgYW5kIHRoZSB0d28g cmVxdWVzdHMgcmFjZSB0byBzZXQgdGhlIHNlcnZpY2UgbGV2ZWwuDQo+Pj4+Pj4+Pj4+Pj4+Pj4g DQo+Pj4+Pj4+Pj4+Pj4+Pj4gSSBhZ3JlZSB3aXRoIEFuZHkuIEl0IG11c3QgYmUgYSB0aWdodCBy YWNlLg0KPj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+PiBJbiBvbmUgY2FwdHVyZSBJIHNl ZSBzb21ldGhpbmcgbGlrZSB0aGlzIGFmdGVyDQo+Pj4+Pj4+Pj4+Pj4+PiB0aGUgc2VydmVyIHJl c3RhcnRzOg0KPj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+PiBTWU4NCj4+Pj4+Pj4+Pj4+ Pj4+IFNZTiwgQUNLDQo+Pj4+Pj4+Pj4+Pj4+PiBBQ0sNCj4+Pj4+Pj4+Pj4+Pj4+IEMgV1JJVEUN Cj4+Pj4+Pj4+Pj4+Pj4+IEMgU0VRVUVOQ0UNCj4+Pj4+Pj4+Pj4+Pj4+IFIgV1JJVEUgLT4gQ1RY X1BST0JMRU0NCj4+Pj4+Pj4+Pj4+Pj4+IFIgU0VRVUVOQ0UgLT4gQ1RYX1BST0JMRU0NCj4+Pj4+ Pj4+Pj4+Pj4+IEMgTlVMTCAoS1JCNV9BUF9SRVEpDQo+Pj4+Pj4+Pj4+Pj4+PiBSIE5VTEwgKEtS QjVfQVBfUkVQKQ0KPj4+Pj4+Pj4+Pj4+Pj4gQyBXUklURQ0KPj4+Pj4+Pj4+Pj4+Pj4gQyBTRVFV RU5DRQ0KPj4+Pj4+Pj4+Pj4+Pj4gUiBXUklURSAtPiBORlM0RVJSX1NUQUxFX1NUQVRFSUQNCj4+ Pj4+Pj4+Pj4+Pj4+IFIgU0VRVUVOQ0UgLT4gQVVUSF9GQUlMRUQNCj4+Pj4+Pj4+Pj4+Pj4+IA0K Pj4+Pj4+Pj4+Pj4+Pj4gQW5keSdzIHRoZW9yeSBuZWF0bHkgZXhwbGFpbnMgdGhpcyBiZWhhdmlv ci4NCj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4gSSBo YXZlIHRyaWVkIHRvIHJlcHJvZHVjZQ0KPj4+Pj4+Pj4+Pj4+Pj4+IHlvdXIgc2NlbmFyaW8gYW5k IGluIG15IHRlc3RzIG9mIHJlYm9vdGluZyB0aGUgc2VydmVyIGFsbCByZWNvdmVyDQo+Pj4+Pj4+ Pj4+Pj4+Pj4gY29ycmVjdGx5LiBJbiBteSBjYXNlLCBpZiBSRU5FVyB3YXMgdGhlIG9uZSBoaXR0 aW5nIHRoZSBBVVRIX0VSUiB0aGVuDQo+Pj4+Pj4+Pj4+Pj4+Pj4gdGhlIG5ldyBjb250ZXh0IGlz IGVzdGFibGlzaGVkIGFuZCB0aGVuIFJFTkVXIHVzaW5nIGludGVncml0eSBzZXJ2aWNlDQo+Pj4+ Pj4+Pj4+Pj4+Pj4gaXMgcmV0cmllZCB3aXRoIHRoZSBuZXcgY29udGV4dCB3aGljaCBnZXRzIEVS Ul9TVEFMRV9DTElFTlRJRCB3aGljaA0KPj4+Pj4+Pj4+Pj4+Pj4+IHRoZW4gY2xpZW50IHJlY292 ZXJzIGZyb20uIElmIGl0J3MgYW4gb3BlcmF0aW9uIChJIGhhdmUgYSBHRVRBVFRSKQ0KPj4+Pj4+ Pj4+Pj4+Pj4+IHRoYXQgZ2V0cyBBVVRIX0VSUiwgdGhlbiBpdCBnZXRzIG5ldyBjb250ZXh0IGFu ZCBpcyByZXRyaWVkIHVzaW5nIG5vbmUNCj4+Pj4+Pj4+Pj4+Pj4+PiBzZXJ2aWNlLiBUaGVuIFJF TkVXIGdldHMgaXRzIG93biBBVVRIX0VSUiBhcyBpdCB1c2VzIGEgZGlmZmVyZW50DQo+Pj4+Pj4+ Pj4+Pj4+Pj4gY29udGV4dCwgYSBuZXcgY29udGV4dCBpcyBnb3R0ZW4sIFJFTkVXIGlzIHJldHJp ZWQgb3ZlciBpbnRlZ3JpdHkgYW5kDQo+Pj4+Pj4+Pj4+Pj4+Pj4gZ2V0cyBFUlJfU1RBTEVfQ0xJ RU5USUQgd2hpY2ggaXQgcmVjb3ZlcnMgZnJvbS4NCj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+Pj4gSWYgb25lIG9wZXJhdGlvbiBpcyBhbGxvd2VkIHRvIGNvbXBsZXRlLCB0aGVuDQo+Pj4+ Pj4+Pj4+Pj4+PiB0aGUgb3RoZXIgd2lsbCBhbHdheXMgcmVjb2duaXplIHRoYXQgYW5vdGhlcg0K Pj4+Pj4+Pj4+Pj4+Pj4gZnJlc2ggR1NTIGNvbnRleHQgaXMgbmVlZGVkLiBJZiB0d28gYXJlIHNl bnQNCj4+Pj4+Pj4+Pj4+Pj4+IGF0IHRoZSBzYW1lIHRpbWUsIHRoZXkgcmFjZSBhbmQgb25lIGFs d2F5cw0KPj4+Pj4+Pj4+Pj4+Pj4gZmFpbHMuDQo+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ Pj4+IEhlbGVuJ3MgdGVzdCBpbmNsdWRlcyBhIHNlY29uZCBpZGxlIG1vdW50IHBvaW50DQo+Pj4+ Pj4+Pj4+Pj4+PiAoc2VjPWtyYjVpKSBhbmQgbWF5YmUgdGhhdCBpcyBuZWVkZWQgdG8gdHJpZ2dl cg0KPj4+Pj4+Pj4+Pj4+Pj4gdGhlIHJhY2U/DQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+ PiBDaHVjaywgYW55IGNoYW5jZSB0byBnZXQgInJwY2RlYnVnIC1tIHJwYyBhdXRoIiBvdXRwdXQg ZHVyaW5nIHRoZQ0KPj4+Pj4+Pj4+Pj4+PiBmYWlsdXJlIChnc3NkIG9wdGlvbmFsbHkpIChpIHJl YWxpemUgdGhhdCBpdCBtaWdodCBhbHRlciB0aGUgdGltaW5ncw0KPj4+Pj4+Pj4+Pj4+PiBhbmQg bm90IGhpdCB0aGUgaXNzdWUgYnV0IHdvcnRoIGEgc2hvdCk/DQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+ Pj4+Pj4+Pj4gSSdtIHN1cmUgdGhhdCdzIGZpbmUuIEFuIGludGVybmFsIHRlc3RlciBoaXQgdGhp cywNCj4+Pj4+Pj4+Pj4+PiBub3QgYSBjdXN0b21lciwgc28gSSB3aWxsIGFzay4NCj4+Pj4+Pj4+ Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBJIGFncmVlLCB0aG91Z2gsIHRoYXQgdGltaW5nIG1pZ2h0IGJl IGEgcHJvYmxlbToNCj4+Pj4+Pj4+Pj4+PiB0aGVzZSBzeXN0ZW1zIGFsbCBoYXZlIHJlYWwgc2Vy aWFsIGNvbnNvbGVzIHZpYQ0KPj4+Pj4+Pj4+Pj4+IGlMT00sIHNvIC92L2wvbSB0cmFmZmljIGRv ZXMgYnJpbmcgZXZlcnl0aGluZyB0bw0KPj4+Pj4+Pj4+Pj4+IGEgc3RhbmRzdGlsbC4NCj4+Pj4+ Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiBNZWFud2hpbGUsIHdoYXQncyB5b3UncmUgb3BpbmlvbiBh Ym91dCBBVVRIX0ZBSUxFRD8NCj4+Pj4+Pj4+Pj4+PiBTaG91bGQgdGhlIHNlcnZlciByZXR1cm4g UlBDU0VDX0dTU19DVFhQUk9CTEVNDQo+Pj4+Pj4+Pj4+Pj4gaW4gdGhpcyBjYXNlIGluc3RlYWQ/ IElmIGl0IGRpZCwgZG8geW91IHRoaW5rDQo+Pj4+Pj4+Pj4+Pj4gdGhlIExpbnV4IGNsaWVudCB3 b3VsZCByZWNvdmVyIGJ5IGNyZWF0aW5nIGENCj4+Pj4+Pj4+Pj4+PiByZXBsYWNlbWVudCBHU1Mg Y29udGV4dD8NCj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4gQWgsIHllcywgSSBlcXVhdGVkIEFV VEhfRkFJTEVEIEFuZCBBVVRIX0VSUk9SIGluIG15IG1pbmQuIElmIGNsaWVudA0KPj4+Pj4+Pj4+ Pj4gcmVjZWl2ZXMgdGhlIHJlYXNvbiBhcyBBVVRIX0ZBSUxFRCBhcyBvcHBvc2UgdG8gQ1RYUFJP QkxFTSBpdCB3aWxsDQo+Pj4+Pj4+Pj4+PiBmYWlsIHdpdGggRUlPIGVycm9yIGFuZCB3aWxsIG5v dCB0cnkgdG8gY3JlYXRlIGEgbmV3IEdTUyBjb250ZXh0LiBTbw0KPj4+Pj4+Pj4+Pj4geWVzLCBJ IGJlbGlldmUgaXQgd291bGQgaGVscCBpZiB0aGUgc2VydmVyIHJldHVybnMgYW55IG9mIHRoZQ0K Pj4+Pj4+Pj4+Pj4gZm9sbG93aW5nIGVycm9yczoNCj4+Pj4+Pj4+Pj4+ICAgICAgICAgICAgY2Fz ZSBSUENfQVVUSF9SRUpFQ1RFRENSRUQ6DQo+Pj4+Pj4+Pj4+PiAgICAgICAgICAgIGNhc2UgUlBD X0FVVEhfUkVKRUNURURWRVJGOg0KPj4+Pj4+Pj4+Pj4gICAgICAgICAgICBjYXNlIFJQQ1NFQ19H U1NfQ1JFRFBST0JMRU06DQo+Pj4+Pj4+Pj4+PiAgICAgICAgICAgIGNhc2UgUlBDU0VDX0dTU19D VFhQUk9CTEVNOg0KPj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+PiB0aGVuIHRoZSBjbGllbnQgd2ls bCByZWNyZWF0ZSB0aGUgY29udGV4dC4NCj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+IEFsc28gaW4g bXkgdGVzdGluZywgSSBjYW4gc2VlIHRoYXQgY3JlZGVudGlhbCBjYWNoZSBpcyBwZXIgZ3NzIGZs YXZvci4NCj4+Pj4+Pj4+Pj4gSnVzdCB0byBjaGVjaywgd2hhdCBrZXJuZWwgdmVyc2lvbiBpcyB0 aGlzIHByb2JsZW0gZW5jb3VudGVyZWQgb24gKEkNCj4+Pj4+Pj4+Pj4ga25vdyB5b3Ugc2FpZCB1 cHN0cmVhbSkgYnV0IEkganVzdCB3YW50IHRvIGRvdWJsZSBjaGVjayBzbyB0aGF0IEkgY2FuDQo+ Pj4+Pj4+Pj4+IGxvb2sgYXQgdGhlIGNvcnJlY3Qgc291cmNlIGNvZGUuDQo+Pj4+Pj4+Pj4gDQo+ Pj4+Pj4+Pj4gdjQuMS4xMiAoc3RhYmxlKSBJIHRoaW5rLg0KPj4+Pj4+Pj4gDQo+Pj4+Pj4+PiBB bHNvLCBjYW4geW91IHNoYXJlIHRoZSBuZXR3b3JrIHRyYWNlPw0KPj4+Pj4+PiANCj4+Pj4+Pj4g SGkgQ2h1Y2ssDQo+Pj4+Pj4+IA0KPj4+Pj4+PiBJIHdhcyBmaW5hbGx5IGFibGUgdG8gcmVwcm9k dWNlIHRoZSBjb25kaXRpb24geW91IHdlcmUgc2VlaW5nIChpLmUuLA0KPj4+Pj4+PiB0aGUgdXNl IG9mIHRoZSBzYW1lIGNvbnRleHQgZm9yIGRpZmZlcmVudCBnc3Mgc2VydmljZXMpLg0KPj4+Pj4+ PiANCj4+Pj4+Pj4gSSBlbmFibGVkIHJwY2RlYnVnIHJwYyBhdXRoIGFuZCBJIGNhbiBzZWUgdGhh dCB0aGUgMm5kIHJlcXVlc3QgZW5kcyB1cA0KPj4+Pj4+PiBmaW5kaW5nIGEgZ3NzX3VwY2FsbCBt ZXNzYWdlIGJlY2F1c2UgaXQncyBqdXN0IG1hdGNoZWQgYnkgdGhlIHVpZC4NCj4+Pj4+Pj4gVGhl cmUgaXMgZXZlbiBhIGNvbW1lbnQgaW4gYXV0aF9nc3MvYXV0aF9nc3MuYyBpbiBnc3NfYWRkX21z ZygpIHNheWluZw0KPj4+Pj4+PiB0aGF0IGlmIHRoZXJlIGlzIHVwY2FsbCBmb3IgYW4gdWlkIHRo ZW4gaXQgd29uJ3QgYWRkIGFub3RoZXIgdXBjYWxsLg0KPj4+Pj4+PiBTbyBJIHRoaW5rIHRoZSBk ZWNpc2lvbiBpcyBtYWRlIHJpZ2h0IHRoZXJlIHRvIHNoYXJlIHRoZSBzYW1lIGNvbnRleHQNCj4+ Pj4+Pj4gbm8gbWF0dGVyIHRoZSBnc3Mgc2VydmljZS4NCj4+Pj4+PiANCj4+Pj4+PiBJZiBJIHVu ZGVyc3RhbmQgY29ycmVjdGx5LCB0aGF0J3MganVzdCB3aGF0IEFuZHkgcHJlZGljdGVkLg0KPj4+ Pj4+IA0KPj4+Pj4+IFRoYXQgY2hlY2sgbmVlZHMgdG8gYmUgY2hhbmdlZCB0byBhbGxvdyBhbm90 aGVyIHVwY2FsbCB0byBiZQ0KPj4+Pj4+IHF1ZXVlZCBpZiB0aGUgVUlEIG1hdGNoZXMgYnV0IHRo ZSBHU1Mgc2VydmljZSBkb2VzIG5vdC4NCj4+Pj4+IA0KPj4+Pj4gWW91IHNob3VsZCBiZSBhYmxl IHRvIHVzZSB0aGUgc2FtZSBjb250ZXh0IHdpdGggZGlmZmVyZW50IHNlcnZpY2VzLg0KPj4+Pj4g DQo+Pj4+PiBBcG9sb2dpZXMsIEkgaGF2ZW4ndCBjYXVnaHQgdXAgd2l0aCB0aGUgd2hvbGUgZGlz Y3Vzc2lvbiBhYm92ZSwgdGhpcyBvbmUNCj4+Pj4+IHBvaW50IGp1c3QganVtcGVkIG91dCBhdCBt ZS4gIElmIHlvdSdyZSB0cnlpbmcgdG8gcmVxdWVzdCBhIHdob2xlIG5ldw0KPj4+Pj4gZ3NzIGNv bnRleHQganVzdCBzbyB5b3UgY2FuIHVzZSwgZS5nLiwgaW50ZWdyaXR5IGluc3RlYWQgb2YgcHJp dmFjeSwNCj4+Pj4+IHRoZW4gc29tZXRoaW5nJ3Mgd3JvbmcuDQo+Pj4+IA0KPj4+PiBUaGUgY2xp ZW50IGNvZGUgaGFzIHNlcGFyYXRlIGdzc19jcmVkIGNhY2hlcyAoYW5kIHNvIHNlcGFyYXRlIGdz c19jb250ZXjigJlzKSBwZXIgZ3NzX2F1dGgsIHdoaWNoIGlzIHBlciBzZXJ2aWNlLiBBRkFJSyB0 aGUgY2xpZW50IGhhcyBhbHdheXMgb2J0YWluZWQgYSBzZXBhcmF0ZSBjb250ZXh0IHBlciBzZXJ2 aWNlIHBlciBzZXJ2ZXIuIFdoaWxlIHdlIGNhbiB1c2UgdGhlIHNhbWUgZ3NzIGNvbnRleHQgd2l0 aCBkaWZmZXJlbnQgc2VydmljZXMsIHRoYXQgaXMgbm90IHRoZSBkZXNpZ24gY2hvaWNlLg0KPj4+ IA0KPj4+IEdpdmVuIHRoZSBjdXJyZW50IGNvZGUsIEknZCBzYXkgdGhhdCBpdCdzIG5vdCBjbGVh ciB3aGF0IHRoZSBkZXNpZ24NCj4+PiBjaG9pY2UgaXMuIFRoZSB1cGNhbGwgY29kZSBzdGF0ZXMg dGhhdCBpdCB3aWxsIG5vdCBkbyBhbm90aGVyIHVwY2FsbA0KPj4+IGZvciBhIGdpdmVuIFVJRCBp ZiBhbm90aGVyIHVwY2FsbCBpcyBhbHJlYWR5IG1hZGUuIFNvIGl0J3MgcHVyZWx5DQo+Pj4gdGlt aW5nIGx1Y2sgdGhhdCB3ZSBoYXZlIHR3byBkaWZmZXJlbnQgY29udGV4dHMgZm9yIHRoZSBtb3Vu dCB3aGVuDQo+Pj4gc2VjPWtyYjUgYW5kIHRoZSBkZWZhdWx0IGZvciBsZWFzZSBvcGVyYXRpb25z IGlzICJpbnRlZ3JpdHkiLiBBbmQgd2UNCj4+PiBkb24ndCBwYXNzIHRoZSBHU1Mgc2VydmljZSBi ZXR3ZWVuIHRoZSBrZXJuZWwgYW5kIGdzc2QgYW5kIHRodXMgbm8gd2F5DQo+Pj4gZm9yIHVzIHRv IHRpZSB0aGUgdXBjYWxsIHRvIHRoZSBzZXJ2aWNlLg0KPj4gDQo+PiBXZWxsLCBJIHdyb3RlIHNv bWUgb2YgdGhlIHVwY2FsbCBjb2RlLCBhbmQgSSB0aG91Z2ggKGluY29ycmVjdGx5LCBJDQo+PiBn dWVzcykgdGhhdCBjbGllbnRzIGNvdWxkIHJldXNlIHRoZSBzYW1lIGNvbnRleHQgd2l0aCBtdWx0 aXBsZSBzZXJ2aWNlcywNCj4+IHNvIHRoZSBjb25mdXNpb24gbWF5IGp1c3QgYmUgbXkgZmF1bHQu ICBTbyB5b3UgdGhpbmsgdGhlIG9ubHkgZml4IGhlcmUNCj4+IGlzIHRvIGtleSB0aGUgdXBjYWxs cw0KPiANCj4gTm8gbmVlZCB0byBjaGFuZ2UgdGhlIHVwY2FsbCwganVzdCBkb27igJkgdCBsZXQg dGhlIDJuZCByZXF1ZXN0IHVzZSB0aGUgbmV3IGRvd25jYWxsIGdzcyBjb250ZXh0IHVubGVzcyB0 aGUgc2VydmljZSBhc3NpZ25lZCB0byBpdCBtYXRjaGVzLiBJIHdvdWxkIHRoaW5rIHlvdSBjb3Vs ZCBzZXQgdGhlIHNlcnZpY2UgaW1tZWRpYXRlbHkgdXBvbiByZWNlaXZpbmcgdGhlIG5ldyBnc3Mg Y29udGV4dCBkb3duIGNhbGwgYW5kIHRoZW4gY2hlY2sgaXQgcHJpb3IgdG8gdGhlIDJuZCByZXF1 ZXN0IHVzZXMgaXQuDQo+IA0KDQpodHRwczovL3Rvb2xzLmlldGYub3JnL2h0bWwvcmZjMjIwMyNz ZWN0aW9uLTUuMi4yIEluIGEgY3JlYXRpb24gcmVxdWVzdCwgdGhlIHNlcV9udW0gYW5kIHNlcnZp Y2UgZmllbGRzIGFyZSB1bmRlZmluZWQgYW5kIGJvdGggbXVzdCBiZSBpZ25vcmVkIGJ5IHRoZSBz ZXJ2ZXIuDQoNCklPVzogdGhlIHNlcnZlciBjYW7igJl0IGxvY2sgdGhlIHNlcnZpY2UgdW50aWwg dGhlIGRhdGEgZXhjaGFuZ2UgcGhhc2UgKGh0dHBzOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9yZmMy MjAzI3NlY3Rpb24tNS4zKS4NCg0KQ2hlZXJzDQogIFRyb25k ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 20:33 ` Trond Myklebust @ 2016-08-03 21:12 ` Adamson, Andy 0 siblings, 0 replies; 25+ messages in thread From: Adamson, Andy @ 2016-08-03 21:12 UTC (permalink / raw) To: Trond Myklebust Cc: Adamson, Andy, Fields Bruce James, Kornievskaia Olga, Lever Chuck, List Linux NFS Mailing DQo+IE9uIEF1ZyAzLCAyMDE2LCBhdCA0OjMzIFBNLCBUcm9uZCBNeWtsZWJ1c3QgPHRyb25kbXlA cHJpbWFyeWRhdGEuY29tPiB3cm90ZToNCj4gDQo+PiANCj4+IE9uIEF1ZyAzLCAyMDE2LCBhdCAx NjoxOCwgQWRhbXNvbiwgQW5keSA8V2lsbGlhbS5BZGFtc29uQG5ldGFwcC5jb20+IHdyb3RlOg0K Pj4gDQo+PiANCj4+PiBPbiBBdWcgMywgMjAxNiwgYXQgNDowNiBQTSwgSi4gQnJ1Y2UgRmllbGRz IDxiZmllbGRzQGZpZWxkc2VzLm9yZz4gd3JvdGU6DQo+Pj4gDQo+Pj4gT24gV2VkLCBBdWcgMDMs IDIwMTYgYXQgMDM6NTY6MzFQTSAtMDQwMCwgT2xnYSBLb3JuaWV2c2thaWEgd3JvdGU6DQo+Pj4+ IE9uIFdlZCwgQXVnIDMsIDIwMTYgYXQgMjo1MyBQTSwgQWRhbXNvbiwgQW5keQ0KPj4+PiA8V2ls bGlhbS5BZGFtc29uQG5ldGFwcC5jb20+IHdyb3RlOg0KPj4+Pj4gDQo+Pj4+Pj4gT24gQXVnIDIs IDIwMTYsIGF0IDI6MDYgUE0sIEouIEJydWNlIEZpZWxkcyA8YmZpZWxkc0BmaWVsZHNlcy5vcmc+ IHdyb3RlOg0KPj4+Pj4+IA0KPj4+Pj4+IE9uIEZyaSwgSnVsIDI5LCAyMDE2IGF0IDEyOjM4OjM0 UE0gLTA0MDAsIENodWNrIExldmVyIHdyb3RlOg0KPj4+Pj4+PiANCj4+Pj4+Pj4+IE9uIEp1bCAy OSwgMjAxNiwgYXQgMTI6MjcgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4g d3JvdGU6DQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+IE9uIE1vbiwgSnVsIDI1LCAyMDE2IGF0IDI6MTgg UE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xvQHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+Pj4g T24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgNTozMiBQTSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVy QG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+ PiBPbiBKdWwgMjEsIDIwMTYsIGF0IDEwOjQ2IFBNLCBPbGdhIEtvcm5pZXZza2FpYSA8YWdsb0B1 bWljaC5lZHU+IHdyb3RlOg0KPj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gT24gVGh1LCBKdWwg MjEsIDIwMTYgYXQgMzo1NCBQTSwgT2xnYSBLb3JuaWV2c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3 cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgMTo1NiBQTSwgQ2h1 Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+Pj4+Pj4+IE9uIEp1bCAyMSwgMjAxNiwgYXQgNjowNCBQTSwgT2xnYSBLb3JuaWV2 c2thaWEgPGFnbG9AdW1pY2guZWR1PiB3cm90ZToNCj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+Pj4gT24gVGh1LCBKdWwgMjEsIDIwMTYgYXQgMjo1NSBBTSwgQ2h1Y2sgTGV2ZXIgPGNodWNr LmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+ Pj4+PiBPbiBKdWwgMjAsIDIwMTYsIGF0IDY6NTYgUE0sIE9sZ2EgS29ybmlldnNrYWlhIDxhZ2xv QHVtaWNoLmVkdT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+PiBP biBXZWQsIEp1bCAyMCwgMjAxNiBhdCA1OjE0IEFNLCBBZGFtc29uLCBBbmR5DQo+Pj4+Pj4+Pj4+ Pj4+Pj4+IDxXaWxsaWFtLkFkYW1zb25AbmV0YXBwLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4+ Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBPbiBKdWwgMTksIDIwMTYsIGF0IDEwOjUxIEFNLCBD aHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4gd3JvdGU6DQo+Pj4+Pj4+Pj4+Pj4+ Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gSGkgQW5keS0NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBUaGFua3MgZm9yIHRha2luZyB0aGUgdGltZSB0byBkaXNjdXNz IHRoaXMgd2l0aCBtZS4gSSd2ZQ0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGNvcGllZCBsaW51eC1uZnMg dG8gbWFrZSB0aGlzIGUtbWFpbCBhbHNvIGFuIHVwc3RyZWFtIGJ1Zw0KPj4+Pj4+Pj4+Pj4+Pj4+ Pj4+IHJlcG9ydC4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBBcyB3 ZSBzYXcgaW4gdGhlIG5ldHdvcmsgY2FwdHVyZSwgcmVjb3Zlcnkgb2YgR1NTIGNvbnRleHRzDQo+ Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gYWZ0ZXIgYSBzZXJ2ZXIgcmVib290IGZhaWxzIGluIGNlcnRhaW4g Y2FzZXMgd2l0aCBORlN2NC4wDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gYW5kIE5GU3Y0LjEgbW91bnQg cG9pbnRzLg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSByZXBy b2R1Y2VyIGlzIGEgc2ltcGxlIHByb2dyYW0gdGhhdCBnZW5lcmF0ZXMgb25lIE5GUw0KPj4+Pj4+ Pj4+Pj4+Pj4+Pj4+IFdSSVRFIHBlcmlvZGljYWxseSwgcnVuIHdoaWxlIHRoZSBzZXJ2ZXIgcmVw ZWF0ZWRseSByZWJvb3RzDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gKG9yIG9uZSBjbHVzdGVyIGhlYWQg ZmFpbHMgb3ZlciB0byB0aGUgb3RoZXIgYW5kIGJhY2spLiBUaGUNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+ PiBnb2FsIG9mIHRoZSByZXByb2R1Y2VyIGlzIHRvIGlkZW50aWZ5IHByb2JsZW1zIHdpdGggc3Rh dGUNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiByZWNvdmVyeSB3aXRob3V0IGEgbG90IG9mIG90aGVyIEkv TyBnb2luZyBvbiB0byBjbHV0dGVyIHVwDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gdGhlIG5ldHdvcmsg Y2FwdHVyZS4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBJbiB0aGUg ZmFpbGluZyBjYXNlLCBzZWM9a3JiNSBpcyBzcGVjaWZpZWQgb24gdGhlIG1vdW50DQo+Pj4+Pj4+ Pj4+Pj4+Pj4+Pj4gcG9pbnQsIGFuZCB0aGUgcmVwcm9kdWNlciBpcyBydW4gYXMgcm9vdC4gV2Un dmUgZm91bmQgdGhpcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGNvbWJpbmF0aW9uIGZhaWxzIHdpdGgg Ym90aCBORlN2NC4wIGFuZCBORlN2NC4xLg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+Pj4+Pj4+IEF0IG1vdW50IHRpbWUsIHRoZSBjbGllbnQgZXN0YWJsaXNoZXMgYSBHU1MgY29u dGV4dCBmb3INCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBsZWFzZSBtYW5hZ2VtZW50IG9wZXJhdGlvbnMs IHdoaWNoIGlzIGJvdW5kIHRvIHRoZSBjbGllbnQncw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IE5GUyBz ZXJ2aWNlIHByaW5jaXBhbCBhbmQgdXNlcyBHU1Mgc2VydmljZSAiaW50ZWdyaXR5LiINCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+PiBDYWxsIHRoaXMgR1NTIGNvbnRleHQgMS4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+ PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBXaGVuIHRoZSByZXByb2R1Y2VyIHN0YXJ0cywgYSBzZWNv bmQgR1NTIGNvbnRleHQgaXMNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBlc3RhYmxpc2hlZCBmb3IgTkZT IG9wZXJhdGlvbnMgYXNzb2NpYXRlZCB3aXRoIHRoYXQgdXNlci4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+ PiBTaW5jZSB0aGUgcmVwcm9kdWNlciBpcyBydW5uaW5nIGFzIHJvb3QsIHRoaXMgY29udGV4dCBp cw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGFsc28gYm91bmQgdG8gdGhlIGNsaWVudCdzIE5GUyBzZXJ2 aWNlIHByaW5jaXBhbCwgYnV0IGl0DQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gdXNlcyB0aGUgR1NTIHNl cnZpY2UgIm5vbmUiIChyZWZsZWN0aW5nIHRoZSBleHBsaWNpdA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+ IHJlcXVlc3QgZm9yICJzZWM9a3JiNSIpLiBDYWxsIHRoaXMgR1NTIGNvbnRleHQgMi4NCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBBZnRlciB0aGUgc2VydmVyIHJlYm9v dHMsIHRoZSBjbGllbnQgcmUtZXN0YWJsaXNoZXMgYSBUQ1ANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBj b25uZWN0aW9uIHdpdGggdGhlIHNlcnZlciwgYW5kIHBlcmZvcm1zIGEgUkVORVcNCj4+Pj4+Pj4+ Pj4+Pj4+Pj4+PiBvcGVyYXRpb24gdXNpbmcgY29udGV4dCAxLiBUaGFua3MgdG8gdGhlIHNlcnZl ciByZWJvb3QsDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gY29udGV4dHMgMSBhbmQgMiBhcmUgbm93IHN0 YWxlLiBUaGUgc2VydmVyIHRodXMgcmVqZWN0cw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHRoZSBSUEMg d2l0aCBSUENTRUNfR1NTX0NUWFBST0JMRU0uDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+ Pj4+Pj4+Pj4+Pj4gVGhlIGNsaWVudCBwZXJmb3JtcyBhIEdTU19JTklUX1NFQ19DT05URVhUIHZp YSBhbiBORlN2NA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IE5VTEwgb3BlcmF0aW9uLiBDYWxsIHRoaXMg R1NTIGNvbnRleHQgMy4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBJ bnRlcmVzdGluZ2x5LCB0aGUgY2xpZW50IGRvZXMgbm90IHJlc2VuZCB0aGUgUkVORVcNCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+PiBvcGVyYXRpb24gYXQgdGhpcyBwb2ludCAoaWYgaXQgZGlkLCB3ZSB3b3Vs ZG4ndCBzZWUgdGhpcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHByb2JsZW0gYXQgYWxsKS4NCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBUaGUgY2xpZW50IHRoZW4gYXR0ZW1w dHMgdG8gcmVzdW1lIHRoZSByZXByb2R1Y2VyIHdvcmtsb2FkLg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+ IEl0IHNlbmRzIGFuIE5GU3Y0IFdSSVRFIG9wZXJhdGlvbiwgdXNpbmcgdGhlIGZpcnN0IGF2YWls YWJsZQ0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IEdTUyBjb250ZXh0IGluIFVJRCAwJ3MgY3JlZGVudGlh bCBjYWNoZSwgd2hpY2ggaXMgY29udGV4dCAzLA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGFscmVhZHkg Ym91bmQgdG8gdGhlIGNsaWVudCdzIE5GUyBzZXJ2aWNlIHByaW5jaXBhbC4gQnV0IEdTUw0KPj4+ Pj4+Pj4+Pj4+Pj4+Pj4+IHNlcnZpY2UgIm5vbmUiIGlzIHVzZWQgZm9yIHRoaXMgb3BlcmF0aW9u LCBzaW5jZSBpdCBpcyBvbg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGJlaGFsZiBvZiB0aGUgbW91bnQg d2hlcmUgc2VjPWtyYjUgd2FzIHNwZWNpZmllZC4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+ Pj4+Pj4+Pj4+Pj4+PiBUaGUgUlBDIGlzIGFjY2VwdGVkLCBidXQgdGhlIHNlcnZlciByZXBvcnRz DQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gTkZTNEVSUl9TVEFMRV9TVEFURUlELCBzaW5jZSBpdCBoYXMg cmVjZW50bHkgcmVib290ZWQuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+ Pj4gVGhlIGNsaWVudCByZXNwb25kcyBieSBhdHRlbXB0aW5nIHN0YXRlIHJlY292ZXJ5LiBUaGUN Cj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBmaXJzdCBvcGVyYXRpb24gaXQgdHJpZXMgaXMgYW5vdGhlciBS RU5FVy4gU2luY2UgdGhpcyBpcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGEgbGVhc2UgbWFuYWdlbWVu dCBvcGVyYXRpb24sIHRoZSBjbGllbnQgbG9va3MgaW4gVUlEIDAncw0KPj4+Pj4+Pj4+Pj4+Pj4+ Pj4+IGNyZWRlbnRpYWwgY2FjaGUgYWdhaW4gYW5kIGZpbmRzIHRoZSByZWNlbnRseSBlc3RhYmxp c2hlZA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGNvbnRleHQgMy4gSXQgdHJpZXMgdGhlIFJFTkVXIG9w ZXJhdGlvbiB1c2luZyBHU1MgY29udGV4dA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IDMgd2l0aCBHU1Mg c2VydmljZSAiaW50ZWdyaXR5LiINCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+ Pj4+PiBUaGUgc2VydmVyIHJlamVjdHMgdGhlIFJFTkVXIFJQQyB3aXRoIEFVVEhfRkFJTEVELCBh bmQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiB0aGUgY2xpZW50IHJlcG9ydHMgdGhhdCAiY2hlY2sgbGVh c2UgZmFpbGVkIiBhbmQNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiB0ZXJtaW5hdGVzIHN0YXRlIHJlY292 ZXJ5Lg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IFRoZSBjbGllbnQg cmUtZHJpdmVzIHRoZSBXUklURSBvcGVyYXRpb24gd2l0aCB0aGUgc3RhbGUNCj4+Pj4+Pj4+Pj4+ Pj4+Pj4+PiBzdGF0ZWlkIHdpdGggcHJlZGljdGFibGUgcmVzdWx0cy4gVGhlIGNsaWVudCBhZ2Fp biB0cmllcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHRvIHJlY292ZXIgc3RhdGUgYnkgc2VuZGluZyBh IFJFTkVXLCBhbmQgc3RpbGwgdXNlcyB0aGUNCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBzYW1lIEdTUyBj b250ZXh0IDMgd2l0aCBzZXJ2aWNlICJpbnRlZ3JpdHkiIGFuZCBnZXRzIHRoZQ0KPj4+Pj4+Pj4+ Pj4+Pj4+Pj4+IHNhbWUgcmVzdWx0LiBBIChwZXJoYXBzIHNsb3ctbW90aW9uKSBTVEFMRV9TVEFU RUlEIGxvb3ANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+PiBlbnN1ZXMsIGFuZCB0aGUgY2xpZW50IG1vdW50 IHBvaW50IGlzIGRlYWRsb2NrZWQuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+ Pj4+Pj4gWW91ciBhbmFseXNpcyB3YXMgdGhhdCBiZWNhdXNlIHRoZSByZXByb2R1Y2VyIGlzIHJ1 biBhcw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IHJvb3QsIGJvdGggdGhlIHJlcHJvZHVjZXIncyBJL08g b3BlcmF0aW9ucywgYW5kIGxlYXNlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gbWFuYWdlbWVudCBvcGVy YXRpb25zLCBhdHRlbXB0IHRvIHVzZSB0aGUgc2FtZSBHU1MgY29udGV4dA0KPj4+Pj4+Pj4+Pj4+ Pj4+Pj4+IGluIFVJRCAwJ3MgY3JlZGVudGlhbCBjYWNoZSwgYnV0IGVhY2ggdXNlcyBkaWZmZXJl bnQgR1NTDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gc2VydmljZXMuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiAN Cj4+Pj4+Pj4+Pj4+Pj4+Pj4+IEFzIFJGQzIyMDMgc3RhdGVzLCAiSW4gYSBjcmVhdGlvbiByZXF1 ZXN0LCB0aGUgc2VxX251bSBhbmQgc2VydmljZSBmaWVsZHMgYXJlIHVuZGVmaW5lZCBhbmQgYm90 aCBtdXN0IGJlIGlnbm9yZWQgYnkgdGhlIHNlcnZlcuKAnQ0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gU28g YSBjb250ZXh0IGNyZWF0aW9uIHJlcXVlc3Qgd2hpbGUga2lja2VkIG9mZiBieSBhbiBvcGVyYXRp b24gd2l0aCBhIHNlcnZpY2UgYXR0YWNoZWQgKGUuZy4gV1JJVEUgdXNlcyBycGNfZ3NzX3N2Y19u b25lIGFuZCBSRU5FVyB1c2VzIHJwY19nc3Nfc3ZjX2ludGVncml0eSksIGNhbiBiZSB1c2VkIGJ5 IGVpdGhlciBzZXJ2aWNlIGxldmVsLg0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4gQUZBSUNTIGEgc2luZ2xl IEdTUyBjb250ZXh0IGNvdWxkIGluIHRoZW9yeSBiZSB1c2VkIGZvciBhbGwgc2VydmljZSBsZXZl bHMsIGJ1dCBpbiBwcmFjdGljZSwgR1NTIGNvbnRleHRzIGFyZSByZXN0cmljdGVkIHRvIGEgc2Vy dmljZSBsZXZlbCAoYnkgY2xpZW50PyBieSBzZXJ2ZXI/ICkgb25jZSB0aGV5IGFyZSB1c2VkLg0K Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+ PiBUaGUga2V5IGlzc3VlIHNlZW1zIHRvIGJlIHdoeSwgd2hlbiB0aGUgbW91bnQNCj4+Pj4+Pj4+ Pj4+Pj4+Pj4+PiBpcyBmaXJzdCBlc3RhYmxpc2hlZCwgdGhlIGNsaWVudCBpcyBjb3JyZWN0bHkg YWJsZSB0bw0KPj4+Pj4+Pj4+Pj4+Pj4+Pj4+IGVzdGFibGlzaCB0d28gc2VwYXJhdGUgR1NTIGNv bnRleHRzIGZvciBVSUQgMDsgYnV0IGFmdGVyDQo+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4gYSBzZXJ2ZXIg cmVib290LCB0aGUgY2xpZW50IGF0dGVtcHRzIHRvIHVzZSB0aGUgc2FtZSBHU1MNCj4+Pj4+Pj4+ Pj4+Pj4+Pj4+PiBjb250ZXh0IHdpdGggdHdvIGRpZmZlcmVudCBHU1Mgc2VydmljZXMuDQo+Pj4+ Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+Pj4+IEkgc3BlY3VsYXRlIHRoYXQgaXQgaXMg YSByYWNlIGJldHdlZW4gdGhlIFdSSVRFIGFuZCB0aGUgUkVORVcgdG8gdXNlIHRoZSBzYW1lIG5l d2x5IGNyZWF0ZWQgR1NTIGNvbnRleHQgdGhhdCBoYXMgbm90IGJlZW4gdXNlZCB5ZXQsIGFuZCBz byBoYXMgbm8gYXNzaWduZWQgc2VydmljZSBsZXZlbCwgYW5kIHRoZSB0d28gcmVxdWVzdHMgcmFj ZSB0byBzZXQgdGhlIHNlcnZpY2UgbGV2ZWwuDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+ Pj4+Pj4+PiBJIGFncmVlIHdpdGggQW5keS4gSXQgbXVzdCBiZSBhIHRpZ2h0IHJhY2UuDQo+Pj4+ Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4gSW4gb25lIGNhcHR1cmUgSSBzZWUgc29tZXRo aW5nIGxpa2UgdGhpcyBhZnRlcg0KPj4+Pj4+Pj4+Pj4+Pj4+IHRoZSBzZXJ2ZXIgcmVzdGFydHM6 DQo+Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4gU1lODQo+Pj4+Pj4+Pj4+Pj4+Pj4g U1lOLCBBQ0sNCj4+Pj4+Pj4+Pj4+Pj4+PiBBQ0sNCj4+Pj4+Pj4+Pj4+Pj4+PiBDIFdSSVRFDQo+ Pj4+Pj4+Pj4+Pj4+Pj4gQyBTRVFVRU5DRQ0KPj4+Pj4+Pj4+Pj4+Pj4+IFIgV1JJVEUgLT4gQ1RY X1BST0JMRU0NCj4+Pj4+Pj4+Pj4+Pj4+PiBSIFNFUVVFTkNFIC0+IENUWF9QUk9CTEVNDQo+Pj4+ Pj4+Pj4+Pj4+Pj4gQyBOVUxMIChLUkI1X0FQX1JFUSkNCj4+Pj4+Pj4+Pj4+Pj4+PiBSIE5VTEwg KEtSQjVfQVBfUkVQKQ0KPj4+Pj4+Pj4+Pj4+Pj4+IEMgV1JJVEUNCj4+Pj4+Pj4+Pj4+Pj4+PiBD IFNFUVVFTkNFDQo+Pj4+Pj4+Pj4+Pj4+Pj4gUiBXUklURSAtPiBORlM0RVJSX1NUQUxFX1NUQVRF SUQNCj4+Pj4+Pj4+Pj4+Pj4+PiBSIFNFUVVFTkNFIC0+IEFVVEhfRkFJTEVEDQo+Pj4+Pj4+Pj4+ Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4gQW5keSdzIHRoZW9yeSBuZWF0bHkgZXhwbGFpbnMgdGhp cyBiZWhhdmlvci4NCj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+ Pj4+Pj4+Pj4gSSBoYXZlIHRyaWVkIHRvIHJlcHJvZHVjZQ0KPj4+Pj4+Pj4+Pj4+Pj4+PiB5b3Vy IHNjZW5hcmlvIGFuZCBpbiBteSB0ZXN0cyBvZiByZWJvb3RpbmcgdGhlIHNlcnZlciBhbGwgcmVj b3Zlcg0KPj4+Pj4+Pj4+Pj4+Pj4+PiBjb3JyZWN0bHkuIEluIG15IGNhc2UsIGlmIFJFTkVXIHdh cyB0aGUgb25lIGhpdHRpbmcgdGhlIEFVVEhfRVJSIHRoZW4NCj4+Pj4+Pj4+Pj4+Pj4+Pj4gdGhl IG5ldyBjb250ZXh0IGlzIGVzdGFibGlzaGVkIGFuZCB0aGVuIFJFTkVXIHVzaW5nIGludGVncml0 eSBzZXJ2aWNlDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IGlzIHJldHJpZWQgd2l0aCB0aGUgbmV3IGNvbnRl eHQgd2hpY2ggZ2V0cyBFUlJfU1RBTEVfQ0xJRU5USUQgd2hpY2gNCj4+Pj4+Pj4+Pj4+Pj4+Pj4g dGhlbiBjbGllbnQgcmVjb3ZlcnMgZnJvbS4gSWYgaXQncyBhbiBvcGVyYXRpb24gKEkgaGF2ZSBh IEdFVEFUVFIpDQo+Pj4+Pj4+Pj4+Pj4+Pj4+IHRoYXQgZ2V0cyBBVVRIX0VSUiwgdGhlbiBpdCBn ZXRzIG5ldyBjb250ZXh0IGFuZCBpcyByZXRyaWVkIHVzaW5nIG5vbmUNCj4+Pj4+Pj4+Pj4+Pj4+ Pj4gc2VydmljZS4gVGhlbiBSRU5FVyBnZXRzIGl0cyBvd24gQVVUSF9FUlIgYXMgaXQgdXNlcyBh IGRpZmZlcmVudA0KPj4+Pj4+Pj4+Pj4+Pj4+PiBjb250ZXh0LCBhIG5ldyBjb250ZXh0IGlzIGdv dHRlbiwgUkVORVcgaXMgcmV0cmllZCBvdmVyIGludGVncml0eSBhbmQNCj4+Pj4+Pj4+Pj4+Pj4+ Pj4gZ2V0cyBFUlJfU1RBTEVfQ0xJRU5USUQgd2hpY2ggaXQgcmVjb3ZlcnMgZnJvbS4NCj4+Pj4+ Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+Pj4+PiBJZiBvbmUgb3BlcmF0aW9uIGlzIGFsbG93ZWQg dG8gY29tcGxldGUsIHRoZW4NCj4+Pj4+Pj4+Pj4+Pj4+PiB0aGUgb3RoZXIgd2lsbCBhbHdheXMg cmVjb2duaXplIHRoYXQgYW5vdGhlcg0KPj4+Pj4+Pj4+Pj4+Pj4+IGZyZXNoIEdTUyBjb250ZXh0 IGlzIG5lZWRlZC4gSWYgdHdvIGFyZSBzZW50DQo+Pj4+Pj4+Pj4+Pj4+Pj4gYXQgdGhlIHNhbWUg dGltZSwgdGhleSByYWNlIGFuZCBvbmUgYWx3YXlzDQo+Pj4+Pj4+Pj4+Pj4+Pj4gZmFpbHMuDQo+ Pj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+Pj4gSGVsZW4ncyB0ZXN0IGluY2x1ZGVzIGEg c2Vjb25kIGlkbGUgbW91bnQgcG9pbnQNCj4+Pj4+Pj4+Pj4+Pj4+PiAoc2VjPWtyYjVpKSBhbmQg bWF5YmUgdGhhdCBpcyBuZWVkZWQgdG8gdHJpZ2dlcg0KPj4+Pj4+Pj4+Pj4+Pj4+IHRoZSByYWNl Pw0KPj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+PiBDaHVjaywgYW55IGNoYW5jZSB0byBn ZXQgInJwY2RlYnVnIC1tIHJwYyBhdXRoIiBvdXRwdXQgZHVyaW5nIHRoZQ0KPj4+Pj4+Pj4+Pj4+ Pj4gZmFpbHVyZSAoZ3NzZCBvcHRpb25hbGx5KSAoaSByZWFsaXplIHRoYXQgaXQgbWlnaHQgYWx0 ZXIgdGhlIHRpbWluZ3MNCj4+Pj4+Pj4+Pj4+Pj4+IGFuZCBub3QgaGl0IHRoZSBpc3N1ZSBidXQg d29ydGggYSBzaG90KT8NCj4+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4+IEknbSBzdXJlIHRo YXQncyBmaW5lLiBBbiBpbnRlcm5hbCB0ZXN0ZXIgaGl0IHRoaXMsDQo+Pj4+Pj4+Pj4+Pj4+IG5v dCBhIGN1c3RvbWVyLCBzbyBJIHdpbGwgYXNrLg0KPj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+ Pj4gSSBhZ3JlZSwgdGhvdWdoLCB0aGF0IHRpbWluZyBtaWdodCBiZSBhIHByb2JsZW06DQo+Pj4+ Pj4+Pj4+Pj4+IHRoZXNlIHN5c3RlbXMgYWxsIGhhdmUgcmVhbCBzZXJpYWwgY29uc29sZXMgdmlh DQo+Pj4+Pj4+Pj4+Pj4+IGlMT00sIHNvIC92L2wvbSB0cmFmZmljIGRvZXMgYnJpbmcgZXZlcnl0 aGluZyB0bw0KPj4+Pj4+Pj4+Pj4+PiBhIHN0YW5kc3RpbGwuDQo+Pj4+Pj4+Pj4+Pj4+IA0KPj4+ Pj4+Pj4+Pj4+PiBNZWFud2hpbGUsIHdoYXQncyB5b3UncmUgb3BpbmlvbiBhYm91dCBBVVRIX0ZB SUxFRD8NCj4+Pj4+Pj4+Pj4+Pj4gU2hvdWxkIHRoZSBzZXJ2ZXIgcmV0dXJuIFJQQ1NFQ19HU1Nf Q1RYUFJPQkxFTQ0KPj4+Pj4+Pj4+Pj4+PiBpbiB0aGlzIGNhc2UgaW5zdGVhZD8gSWYgaXQgZGlk LCBkbyB5b3UgdGhpbmsNCj4+Pj4+Pj4+Pj4+Pj4gdGhlIExpbnV4IGNsaWVudCB3b3VsZCByZWNv dmVyIGJ5IGNyZWF0aW5nIGENCj4+Pj4+Pj4+Pj4+Pj4gcmVwbGFjZW1lbnQgR1NTIGNvbnRleHQ/ DQo+Pj4+Pj4+Pj4+Pj4gDQo+Pj4+Pj4+Pj4+Pj4gQWgsIHllcywgSSBlcXVhdGVkIEFVVEhfRkFJ TEVEIEFuZCBBVVRIX0VSUk9SIGluIG15IG1pbmQuIElmIGNsaWVudA0KPj4+Pj4+Pj4+Pj4+IHJl Y2VpdmVzIHRoZSByZWFzb24gYXMgQVVUSF9GQUlMRUQgYXMgb3Bwb3NlIHRvIENUWFBST0JMRU0g aXQgd2lsbA0KPj4+Pj4+Pj4+Pj4+IGZhaWwgd2l0aCBFSU8gZXJyb3IgYW5kIHdpbGwgbm90IHRy eSB0byBjcmVhdGUgYSBuZXcgR1NTIGNvbnRleHQuIFNvDQo+Pj4+Pj4+Pj4+Pj4geWVzLCBJIGJl bGlldmUgaXQgd291bGQgaGVscCBpZiB0aGUgc2VydmVyIHJldHVybnMgYW55IG9mIHRoZQ0KPj4+ Pj4+Pj4+Pj4+IGZvbGxvd2luZyBlcnJvcnM6DQo+Pj4+Pj4+Pj4+Pj4gICAgICAgICAgIGNhc2Ug UlBDX0FVVEhfUkVKRUNURURDUkVEOg0KPj4+Pj4+Pj4+Pj4+ICAgICAgICAgICBjYXNlIFJQQ19B VVRIX1JFSkVDVEVEVkVSRjoNCj4+Pj4+Pj4+Pj4+PiAgICAgICAgICAgY2FzZSBSUENTRUNfR1NT X0NSRURQUk9CTEVNOg0KPj4+Pj4+Pj4+Pj4+ICAgICAgICAgICBjYXNlIFJQQ1NFQ19HU1NfQ1RY UFJPQkxFTToNCj4+Pj4+Pj4+Pj4+PiANCj4+Pj4+Pj4+Pj4+PiB0aGVuIHRoZSBjbGllbnQgd2ls bCByZWNyZWF0ZSB0aGUgY29udGV4dC4NCj4+Pj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4+Pj4gQWxzbyBp biBteSB0ZXN0aW5nLCBJIGNhbiBzZWUgdGhhdCBjcmVkZW50aWFsIGNhY2hlIGlzIHBlciBnc3Mg Zmxhdm9yLg0KPj4+Pj4+Pj4+Pj4gSnVzdCB0byBjaGVjaywgd2hhdCBrZXJuZWwgdmVyc2lvbiBp cyB0aGlzIHByb2JsZW0gZW5jb3VudGVyZWQgb24gKEkNCj4+Pj4+Pj4+Pj4+IGtub3cgeW91IHNh aWQgdXBzdHJlYW0pIGJ1dCBJIGp1c3Qgd2FudCB0byBkb3VibGUgY2hlY2sgc28gdGhhdCBJIGNh bg0KPj4+Pj4+Pj4+Pj4gbG9vayBhdCB0aGUgY29ycmVjdCBzb3VyY2UgY29kZS4NCj4+Pj4+Pj4+ Pj4gDQo+Pj4+Pj4+Pj4+IHY0LjEuMTIgKHN0YWJsZSkgSSB0aGluay4NCj4+Pj4+Pj4+PiANCj4+ Pj4+Pj4+PiBBbHNvLCBjYW4geW91IHNoYXJlIHRoZSBuZXR3b3JrIHRyYWNlPw0KPj4+Pj4+Pj4g DQo+Pj4+Pj4+PiBIaSBDaHVjaywNCj4+Pj4+Pj4+IA0KPj4+Pj4+Pj4gSSB3YXMgZmluYWxseSBh YmxlIHRvIHJlcHJvZHVjZSB0aGUgY29uZGl0aW9uIHlvdSB3ZXJlIHNlZWluZyAoaS5lLiwNCj4+ Pj4+Pj4+IHRoZSB1c2Ugb2YgdGhlIHNhbWUgY29udGV4dCBmb3IgZGlmZmVyZW50IGdzcyBzZXJ2 aWNlcykuDQo+Pj4+Pj4+PiANCj4+Pj4+Pj4+IEkgZW5hYmxlZCBycGNkZWJ1ZyBycGMgYXV0aCBh bmQgSSBjYW4gc2VlIHRoYXQgdGhlIDJuZCByZXF1ZXN0IGVuZHMgdXANCj4+Pj4+Pj4+IGZpbmRp bmcgYSBnc3NfdXBjYWxsIG1lc3NhZ2UgYmVjYXVzZSBpdCdzIGp1c3QgbWF0Y2hlZCBieSB0aGUg dWlkLg0KPj4+Pj4+Pj4gVGhlcmUgaXMgZXZlbiBhIGNvbW1lbnQgaW4gYXV0aF9nc3MvYXV0aF9n c3MuYyBpbiBnc3NfYWRkX21zZygpIHNheWluZw0KPj4+Pj4+Pj4gdGhhdCBpZiB0aGVyZSBpcyB1 cGNhbGwgZm9yIGFuIHVpZCB0aGVuIGl0IHdvbid0IGFkZCBhbm90aGVyIHVwY2FsbC4NCj4+Pj4+ Pj4+IFNvIEkgdGhpbmsgdGhlIGRlY2lzaW9uIGlzIG1hZGUgcmlnaHQgdGhlcmUgdG8gc2hhcmUg dGhlIHNhbWUgY29udGV4dA0KPj4+Pj4+Pj4gbm8gbWF0dGVyIHRoZSBnc3Mgc2VydmljZS4NCj4+ Pj4+Pj4gDQo+Pj4+Pj4+IElmIEkgdW5kZXJzdGFuZCBjb3JyZWN0bHksIHRoYXQncyBqdXN0IHdo YXQgQW5keSBwcmVkaWN0ZWQuDQo+Pj4+Pj4+IA0KPj4+Pj4+PiBUaGF0IGNoZWNrIG5lZWRzIHRv IGJlIGNoYW5nZWQgdG8gYWxsb3cgYW5vdGhlciB1cGNhbGwgdG8gYmUNCj4+Pj4+Pj4gcXVldWVk IGlmIHRoZSBVSUQgbWF0Y2hlcyBidXQgdGhlIEdTUyBzZXJ2aWNlIGRvZXMgbm90Lg0KPj4+Pj4+ IA0KPj4+Pj4+IFlvdSBzaG91bGQgYmUgYWJsZSB0byB1c2UgdGhlIHNhbWUgY29udGV4dCB3aXRo IGRpZmZlcmVudCBzZXJ2aWNlcy4NCj4+Pj4+PiANCj4+Pj4+PiBBcG9sb2dpZXMsIEkgaGF2ZW4n dCBjYXVnaHQgdXAgd2l0aCB0aGUgd2hvbGUgZGlzY3Vzc2lvbiBhYm92ZSwgdGhpcyBvbmUNCj4+ Pj4+PiBwb2ludCBqdXN0IGp1bXBlZCBvdXQgYXQgbWUuICBJZiB5b3UncmUgdHJ5aW5nIHRvIHJl cXVlc3QgYSB3aG9sZSBuZXcNCj4+Pj4+PiBnc3MgY29udGV4dCBqdXN0IHNvIHlvdSBjYW4gdXNl LCBlLmcuLCBpbnRlZ3JpdHkgaW5zdGVhZCBvZiBwcml2YWN5LA0KPj4+Pj4+IHRoZW4gc29tZXRo aW5nJ3Mgd3JvbmcuDQo+Pj4+PiANCj4+Pj4+IFRoZSBjbGllbnQgY29kZSBoYXMgc2VwYXJhdGUg Z3NzX2NyZWQgY2FjaGVzIChhbmQgc28gc2VwYXJhdGUgZ3NzX2NvbnRleOKAmXMpIHBlciBnc3Nf YXV0aCwgd2hpY2ggaXMgcGVyIHNlcnZpY2UuIEFGQUlLIHRoZSBjbGllbnQgaGFzIGFsd2F5cyBv YnRhaW5lZCBhIHNlcGFyYXRlIGNvbnRleHQgcGVyIHNlcnZpY2UgcGVyIHNlcnZlci4gV2hpbGUg d2UgY2FuIHVzZSB0aGUgc2FtZSBnc3MgY29udGV4dCB3aXRoIGRpZmZlcmVudCBzZXJ2aWNlcywg dGhhdCBpcyBub3QgdGhlIGRlc2lnbiBjaG9pY2UuDQo+Pj4+IA0KPj4+PiBHaXZlbiB0aGUgY3Vy cmVudCBjb2RlLCBJJ2Qgc2F5IHRoYXQgaXQncyBub3QgY2xlYXIgd2hhdCB0aGUgZGVzaWduDQo+ Pj4+IGNob2ljZSBpcy4gVGhlIHVwY2FsbCBjb2RlIHN0YXRlcyB0aGF0IGl0IHdpbGwgbm90IGRv IGFub3RoZXIgdXBjYWxsDQo+Pj4+IGZvciBhIGdpdmVuIFVJRCBpZiBhbm90aGVyIHVwY2FsbCBp cyBhbHJlYWR5IG1hZGUuIFNvIGl0J3MgcHVyZWx5DQo+Pj4+IHRpbWluZyBsdWNrIHRoYXQgd2Ug aGF2ZSB0d28gZGlmZmVyZW50IGNvbnRleHRzIGZvciB0aGUgbW91bnQgd2hlbg0KPj4+PiBzZWM9 a3JiNSBhbmQgdGhlIGRlZmF1bHQgZm9yIGxlYXNlIG9wZXJhdGlvbnMgaXMgImludGVncml0eSIu IEFuZCB3ZQ0KPj4+PiBkb24ndCBwYXNzIHRoZSBHU1Mgc2VydmljZSBiZXR3ZWVuIHRoZSBrZXJu ZWwgYW5kIGdzc2QgYW5kIHRodXMgbm8gd2F5DQo+Pj4+IGZvciB1cyB0byB0aWUgdGhlIHVwY2Fs bCB0byB0aGUgc2VydmljZS4NCj4+PiANCj4+PiBXZWxsLCBJIHdyb3RlIHNvbWUgb2YgdGhlIHVw Y2FsbCBjb2RlLCBhbmQgSSB0aG91Z2ggKGluY29ycmVjdGx5LCBJDQo+Pj4gZ3Vlc3MpIHRoYXQg Y2xpZW50cyBjb3VsZCByZXVzZSB0aGUgc2FtZSBjb250ZXh0IHdpdGggbXVsdGlwbGUgc2Vydmlj ZXMsDQo+Pj4gc28gdGhlIGNvbmZ1c2lvbiBtYXkganVzdCBiZSBteSBmYXVsdC4gIFNvIHlvdSB0 aGluayB0aGUgb25seSBmaXggaGVyZQ0KPj4+IGlzIHRvIGtleSB0aGUgdXBjYWxscw0KPj4gDQo+ PiBObyBuZWVkIHRvIGNoYW5nZSB0aGUgdXBjYWxsLCBqdXN0IGRvbuKAmSB0IGxldCB0aGUgMm5k IHJlcXVlc3QgdXNlIHRoZSBuZXcgZG93bmNhbGwgZ3NzIGNvbnRleHQgdW5sZXNzIHRoZSBzZXJ2 aWNlIGFzc2lnbmVkIHRvIGl0IG1hdGNoZXMuIEkgd291bGQgdGhpbmsgeW91IGNvdWxkIHNldCB0 aGUgc2VydmljZSBpbW1lZGlhdGVseSB1cG9uIHJlY2VpdmluZyB0aGUgbmV3IGdzcyBjb250ZXh0 IGRvd24gY2FsbCBhbmQgdGhlbiBjaGVjayBpdCBwcmlvciB0byB0aGUgMm5kIHJlcXVlc3QgdXNl cyBpdC4NCj4+IA0KPiANCj4gaHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzIyMDMjc2Vj dGlvbi01LjIuMiBJbiBhIGNyZWF0aW9uIHJlcXVlc3QsIHRoZSBzZXFfbnVtIGFuZCBzZXJ2aWNl IGZpZWxkcyBhcmUgdW5kZWZpbmVkIGFuZCBib3RoIG11c3QgYmUgaWdub3JlZCBieSB0aGUgc2Vy dmVyLg0KPiANCj4gSU9XOiB0aGUgc2VydmVyIGNhbuKAmXQgbG9jayB0aGUgc2VydmljZSB1bnRp bCB0aGUgZGF0YSBleGNoYW5nZSBwaGFzZSAoaHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3Jm YzIyMDMjc2VjdGlvbi01LjMpLg0KDQpZZXMsIHRoYXQgaXMgdHJ1ZS4gSSBkb27igJl0IHNlZSBo b3cgdGhhdCBjaGFuZ2VzIHdoYXQgSSBwcm9wb3NlZC4gT25jZSB0aGUgY2xpZW50IGNob29zZXMg dG8gdXNlIGEgZ3NzIGNvbnRleHQgZm9yIGEgc2VydmljZSBsZXZlbCwgaXQgc2hvdWxkIG5vdCB1 c2UgdGhlIHNhbWUgY29udGV4dCBmb3IgYW5vdGhlciBzZXJ2aWNlIGxldmVsIGFzIGl0IGlzIGFs bW9zdCBndWFyYW50ZWVkIHRvIGZhaWwuDQoNCuKAlD5BbmR5DQoNCj4gDQo+IENoZWVycw0KPiAg VHJvbmQNCg0K ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-02 18:06 ` J. Bruce Fields 2016-08-03 18:53 ` Adamson, Andy @ 2016-08-03 19:14 ` Chuck Lever 2016-08-03 19:34 ` J. Bruce Fields 1 sibling, 1 reply; 25+ messages in thread From: Chuck Lever @ 2016-08-03 19:14 UTC (permalink / raw) To: J. Bruce Fields; +Cc: Olga Kornievskaia, Adamson, Andy, Linux NFS Mailing List > On Aug 2, 2016, at 2:06 PM, bfields@fieldses.org wrote: > > On Fri, Jul 29, 2016 at 12:38:34PM -0400, Chuck Lever wrote: >> >>> On Jul 29, 2016, at 12:27 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>> >>> On Mon, Jul 25, 2016 at 2:18 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>> On Thu, Jul 21, 2016 at 5:32 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>> >>>>> >>>>>> On Jul 21, 2016, at 10:46 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>> >>>>>>> On Thu, Jul 21, 2016 at 3:54 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>> On Thu, Jul 21, 2016 at 1:56 PM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>> >>>>>>>>> On Jul 21, 2016, at 6:04 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>> >>>>>>>>> On Thu, Jul 21, 2016 at 2:55 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>> >>>>>>>>>>> On Jul 20, 2016, at 6:56 PM, Olga Kornievskaia <aglo@umich.edu> wrote: >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 20, 2016 at 5:14 AM, Adamson, Andy >>>>>>>>>>> <William.Adamson@netapp.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Jul 19, 2016, at 10:51 AM, Chuck Lever <chuck.lever@oracle.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Andy- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for taking the time to discuss this with me. I've >>>>>>>>>>>>> copied linux-nfs to make this e-mail also an upstream bug >>>>>>>>>>>>> report. >>>>>>>>>>>>> >>>>>>>>>>>>> As we saw in the network capture, recovery of GSS contexts >>>>>>>>>>>>> after a server reboot fails in certain cases with NFSv4.0 >>>>>>>>>>>>> and NFSv4.1 mount points. >>>>>>>>>>>>> >>>>>>>>>>>>> The reproducer is a simple program that generates one NFS >>>>>>>>>>>>> WRITE periodically, run while the server repeatedly reboots >>>>>>>>>>>>> (or one cluster head fails over to the other and back). The >>>>>>>>>>>>> goal of the reproducer is to identify problems with state >>>>>>>>>>>>> recovery without a lot of other I/O going on to clutter up >>>>>>>>>>>>> the network capture. >>>>>>>>>>>>> >>>>>>>>>>>>> In the failing case, sec=krb5 is specified on the mount >>>>>>>>>>>>> point, and the reproducer is run as root. We've found this >>>>>>>>>>>>> combination fails with both NFSv4.0 and NFSv4.1. >>>>>>>>>>>>> >>>>>>>>>>>>> At mount time, the client establishes a GSS context for >>>>>>>>>>>>> lease management operations, which is bound to the client's >>>>>>>>>>>>> NFS service principal and uses GSS service "integrity." >>>>>>>>>>>>> Call this GSS context 1. >>>>>>>>>>>>> >>>>>>>>>>>>> When the reproducer starts, a second GSS context is >>>>>>>>>>>>> established for NFS operations associated with that user. >>>>>>>>>>>>> Since the reproducer is running as root, this context is >>>>>>>>>>>>> also bound to the client's NFS service principal, but it >>>>>>>>>>>>> uses the GSS service "none" (reflecting the explicit >>>>>>>>>>>>> request for "sec=krb5"). Call this GSS context 2. >>>>>>>>>>>>> >>>>>>>>>>>>> After the server reboots, the client re-establishes a TCP >>>>>>>>>>>>> connection with the server, and performs a RENEW >>>>>>>>>>>>> operation using context 1. Thanks to the server reboot, >>>>>>>>>>>>> contexts 1 and 2 are now stale. The server thus rejects >>>>>>>>>>>>> the RPC with RPCSEC_GSS_CTXPROBLEM. >>>>>>>>>>>>> >>>>>>>>>>>>> The client performs a GSS_INIT_SEC_CONTEXT via an NFSv4 >>>>>>>>>>>>> NULL operation. Call this GSS context 3. >>>>>>>>>>>>> >>>>>>>>>>>>> Interestingly, the client does not resend the RENEW >>>>>>>>>>>>> operation at this point (if it did, we wouldn't see this >>>>>>>>>>>>> problem at all). >>>>>>>>>>>>> >>>>>>>>>>>>> The client then attempts to resume the reproducer workload. >>>>>>>>>>>>> It sends an NFSv4 WRITE operation, using the first available >>>>>>>>>>>>> GSS context in UID 0's credential cache, which is context 3, >>>>>>>>>>>>> already bound to the client's NFS service principal. But GSS >>>>>>>>>>>>> service "none" is used for this operation, since it is on >>>>>>>>>>>>> behalf of the mount where sec=krb5 was specified. >>>>>>>>>>>>> >>>>>>>>>>>>> The RPC is accepted, but the server reports >>>>>>>>>>>>> NFS4ERR_STALE_STATEID, since it has recently rebooted. >>>>>>>>>>>>> >>>>>>>>>>>>> The client responds by attempting state recovery. The >>>>>>>>>>>>> first operation it tries is another RENEW. Since this is >>>>>>>>>>>>> a lease management operation, the client looks in UID 0's >>>>>>>>>>>>> credential cache again and finds the recently established >>>>>>>>>>>>> context 3. It tries the RENEW operation using GSS context >>>>>>>>>>>>> 3 with GSS service "integrity." >>>>>>>>>>>>> >>>>>>>>>>>>> The server rejects the RENEW RPC with AUTH_FAILED, and >>>>>>>>>>>>> the client reports that "check lease failed" and >>>>>>>>>>>>> terminates state recovery. >>>>>>>>>>>>> >>>>>>>>>>>>> The client re-drives the WRITE operation with the stale >>>>>>>>>>>>> stateid with predictable results. The client again tries >>>>>>>>>>>>> to recover state by sending a RENEW, and still uses the >>>>>>>>>>>>> same GSS context 3 with service "integrity" and gets the >>>>>>>>>>>>> same result. A (perhaps slow-motion) STALE_STATEID loop >>>>>>>>>>>>> ensues, and the client mount point is deadlocked. >>>>>>>>>>>>> >>>>>>>>>>>>> Your analysis was that because the reproducer is run as >>>>>>>>>>>>> root, both the reproducer's I/O operations, and lease >>>>>>>>>>>>> management operations, attempt to use the same GSS context >>>>>>>>>>>>> in UID 0's credential cache, but each uses different GSS >>>>>>>>>>>>> services. >>>>>>>>>>>> >>>>>>>>>>>> As RFC2203 states, "In a creation request, the seq_num and service fields are undefined and both must be ignored by the server” >>>>>>>>>>>> So a context creation request while kicked off by an operation with a service attached (e.g. WRITE uses rpc_gss_svc_none and RENEW uses rpc_gss_svc_integrity), can be used by either service level. >>>>>>>>>>>> AFAICS a single GSS context could in theory be used for all service levels, but in practice, GSS contexts are restricted to a service level (by client? by server? ) once they are used. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> The key issue seems to be why, when the mount >>>>>>>>>>>>> is first established, the client is correctly able to >>>>>>>>>>>>> establish two separate GSS contexts for UID 0; but after >>>>>>>>>>>>> a server reboot, the client attempts to use the same GSS >>>>>>>>>>>>> context with two different GSS services. >>>>>>>>>>>> >>>>>>>>>>>> I speculate that it is a race between the WRITE and the RENEW to use the same newly created GSS context that has not been used yet, and so has no assigned service level, and the two requests race to set the service level. >>>>>>>>>>> >>>>>>>>>>> I agree with Andy. It must be a tight race. >>>>>>>>>> >>>>>>>>>> In one capture I see something like this after >>>>>>>>>> the server restarts: >>>>>>>>>> >>>>>>>>>> SYN >>>>>>>>>> SYN, ACK >>>>>>>>>> ACK >>>>>>>>>> C WRITE >>>>>>>>>> C SEQUENCE >>>>>>>>>> R WRITE -> CTX_PROBLEM >>>>>>>>>> R SEQUENCE -> CTX_PROBLEM >>>>>>>>>> C NULL (KRB5_AP_REQ) >>>>>>>>>> R NULL (KRB5_AP_REP) >>>>>>>>>> C WRITE >>>>>>>>>> C SEQUENCE >>>>>>>>>> R WRITE -> NFS4ERR_STALE_STATEID >>>>>>>>>> R SEQUENCE -> AUTH_FAILED >>>>>>>>>> >>>>>>>>>> Andy's theory neatly explains this behavior. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I have tried to reproduce >>>>>>>>>>> your scenario and in my tests of rebooting the server all recover >>>>>>>>>>> correctly. In my case, if RENEW was the one hitting the AUTH_ERR then >>>>>>>>>>> the new context is established and then RENEW using integrity service >>>>>>>>>>> is retried with the new context which gets ERR_STALE_CLIENTID which >>>>>>>>>>> then client recovers from. If it's an operation (I have a GETATTR) >>>>>>>>>>> that gets AUTH_ERR, then it gets new context and is retried using none >>>>>>>>>>> service. Then RENEW gets its own AUTH_ERR as it uses a different >>>>>>>>>>> context, a new context is gotten, RENEW is retried over integrity and >>>>>>>>>>> gets ERR_STALE_CLIENTID which it recovers from. >>>>>>>>>> >>>>>>>>>> If one operation is allowed to complete, then >>>>>>>>>> the other will always recognize that another >>>>>>>>>> fresh GSS context is needed. If two are sent >>>>>>>>>> at the same time, they race and one always >>>>>>>>>> fails. >>>>>>>>>> >>>>>>>>>> Helen's test includes a second idle mount point >>>>>>>>>> (sec=krb5i) and maybe that is needed to trigger >>>>>>>>>> the race? >>>>>>>>> >>>>>>>>> Chuck, any chance to get "rpcdebug -m rpc auth" output during the >>>>>>>>> failure (gssd optionally) (i realize that it might alter the timings >>>>>>>>> and not hit the issue but worth a shot)? >>>>>>>> >>>>>>>> I'm sure that's fine. An internal tester hit this, >>>>>>>> not a customer, so I will ask. >>>>>>>> >>>>>>>> I agree, though, that timing might be a problem: >>>>>>>> these systems all have real serial consoles via >>>>>>>> iLOM, so /v/l/m traffic does bring everything to >>>>>>>> a standstill. >>>>>>>> >>>>>>>> Meanwhile, what's you're opinion about AUTH_FAILED? >>>>>>>> Should the server return RPCSEC_GSS_CTXPROBLEM >>>>>>>> in this case instead? If it did, do you think >>>>>>>> the Linux client would recover by creating a >>>>>>>> replacement GSS context? >>>>>>> >>>>>>> Ah, yes, I equated AUTH_FAILED And AUTH_ERROR in my mind. If client >>>>>>> receives the reason as AUTH_FAILED as oppose to CTXPROBLEM it will >>>>>>> fail with EIO error and will not try to create a new GSS context. So >>>>>>> yes, I believe it would help if the server returns any of the >>>>>>> following errors: >>>>>>> case RPC_AUTH_REJECTEDCRED: >>>>>>> case RPC_AUTH_REJECTEDVERF: >>>>>>> case RPCSEC_GSS_CREDPROBLEM: >>>>>>> case RPCSEC_GSS_CTXPROBLEM: >>>>>>> >>>>>>> then the client will recreate the context. >>>>>> >>>>>> Also in my testing, I can see that credential cache is per gss flavor. >>>>>> Just to check, what kernel version is this problem encountered on (I >>>>>> know you said upstream) but I just want to double check so that I can >>>>>> look at the correct source code. >>>>> >>>>> v4.1.12 (stable) I think. >>>> >>>> Also, can you share the network trace? >>> >>> Hi Chuck, >>> >>> I was finally able to reproduce the condition you were seeing (i.e., >>> the use of the same context for different gss services). >>> >>> I enabled rpcdebug rpc auth and I can see that the 2nd request ends up >>> finding a gss_upcall message because it's just matched by the uid. >>> There is even a comment in auth_gss/auth_gss.c in gss_add_msg() saying >>> that if there is upcall for an uid then it won't add another upcall. >>> So I think the decision is made right there to share the same context >>> no matter the gss service. >> >> If I understand correctly, that's just what Andy predicted. >> >> That check needs to be changed to allow another upcall to be >> queued if the UID matches but the GSS service does not. > > You should be able to use the same context with different services. > > Apologies, I haven't caught up with the whole discussion above, this one > point just jumped out at me. If you're trying to request a whole new > gss context just so you can use, e.g., integrity instead of privacy, > then something's wrong. Hi Bruce- As I understand it, GSS contexts are fungible until they have been used. On first use, the context is bound to a particular service. Subsequently it cannot be used with another service. The Solaris server seems to expect that separate GSS contexts are needed when the same UID employs different GSS services. If Solaris is wrong about this, can you show me RFC language that specifically allows it? I can take that back to the Solaris developers. Otherwise, it seems that the Linux client also believes separate contexts are necessary: two contexts are set up for UID 0 after the initial mount processing is complete. -- Chuck Lever ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Problem re-establishing GSS contexts after a server reboot 2016-08-03 19:14 ` Chuck Lever @ 2016-08-03 19:34 ` J. Bruce Fields 0 siblings, 0 replies; 25+ messages in thread From: J. Bruce Fields @ 2016-08-03 19:34 UTC (permalink / raw) To: Chuck Lever; +Cc: Olga Kornievskaia, Adamson, Andy, Linux NFS Mailing List On Wed, Aug 03, 2016 at 03:14:21PM -0400, Chuck Lever wrote: > > On Aug 2, 2016, at 2:06 PM, bfields@fieldses.org wrote: > > You should be able to use the same context with different services. > > > > Apologies, I haven't caught up with the whole discussion above, this one > > point just jumped out at me. If you're trying to request a whole new > > gss context just so you can use, e.g., integrity instead of privacy, > > then something's wrong. > > As I understand it, GSS contexts are fungible until they have been > used. On first use, the context is bound to a particular service. > Subsequently it cannot be used with another service. > > The Solaris server seems to expect that separate GSS contexts are > needed when the same UID employs different GSS services. If Solaris > is wrong about this, can you show me RFC language that specifically > allows it? I can take that back to the Solaris developers. No, you're right, apologies; from https://tools.ietf.org/html/rfc2203 Although clients can change the security service and QOP used on a per-request basis, this may not be acceptable to all RPC services; some RPC services may "lock" the data exchange phase into using the QOP and service used on the first data exchange message. --b. ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2016-08-03 21:19 UTC | newest] Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-07-19 14:51 Problem re-establishing GSS contexts after a server reboot Chuck Lever 2016-07-20 9:14 ` Adamson, Andy 2016-07-20 16:56 ` Olga Kornievskaia 2016-07-21 6:55 ` Chuck Lever 2016-07-21 16:04 ` Olga Kornievskaia 2016-07-21 17:56 ` Chuck Lever 2016-07-21 19:54 ` Olga Kornievskaia 2016-07-21 20:46 ` Olga Kornievskaia 2016-07-21 21:32 ` Chuck Lever 2016-07-25 18:18 ` Olga Kornievskaia 2016-07-29 16:27 ` Olga Kornievskaia 2016-07-29 16:38 ` Chuck Lever 2016-07-29 17:07 ` Adamson, Andy 2016-07-29 17:32 ` Adamson, Andy 2016-07-29 22:24 ` Olga Kornievskaia 2016-08-02 18:06 ` J. Bruce Fields 2016-08-03 18:53 ` Adamson, Andy 2016-08-03 19:56 ` Olga Kornievskaia 2016-08-03 20:06 ` J. Bruce Fields 2016-08-03 20:11 ` Olga Kornievskaia 2016-08-03 20:18 ` Adamson, Andy 2016-08-03 20:33 ` Trond Myklebust 2016-08-03 21:12 ` Adamson, Andy 2016-08-03 19:14 ` Chuck Lever 2016-08-03 19:34 ` J. Bruce Fields
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.