From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Martin Subject: [PATCH v5 13/30] arm64/sve: Core task context handling Date: Tue, 31 Oct 2017 15:51:05 +0000 Message-ID: <1509465082-30427-14-git-send-email-Dave.Martin@arm.com> References: <1509465082-30427-1-git-send-email-Dave.Martin@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1509465082-30427-1-git-send-email-Dave.Martin@arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: linux-arm-kernel@lists.infradead.org Cc: linux-arch@vger.kernel.org, Okamoto Takayuki , libc-alpha@sourceware.org, Ard Biesheuvel , Szabolcs Nagy , Catalin Marinas , Will Deacon , kvmarm@lists.cs.columbia.edu List-Id: linux-arch.vger.kernel.org VGhpcyBwYXRjaCBhZGRzIHRoZSBjb3JlIHN1cHBvcnQgZm9yIHN3aXRjaGluZyBhbmQgbWFuYWdp bmcgdGhlIFNWRQphcmNoaXRlY3R1cmFsIHN0YXRlIG9mIHVzZXIgdGFza3MuCgpDYWxscyB0byB0 aGUgZXhpc3RpbmcgRlBTSU1EIGxvdy1sZXZlbCBzYXZlL3Jlc3RvcmUgZnVuY3Rpb25zIGFyZQpm YWN0b3JlZCBvdXQgYXMgbmV3IGZ1bmN0aW9ucyB0YXNrX2Zwc2ltZF97c2F2ZSxsb2FkfSgpLCBz aW5jZSBTVkUKbm93IGR5bmFtaWNhbGx5IG1heSBvciBtYXkgbm90IG5lZWQgdG8gYmUgaGFuZGxl ZCBhdCB0aGVzZSBwb2ludHMKZGVwZW5kaW5nIG9uIHRoZSBrZXJuZWwgY29uZmlndXJhdGlvbiwg aGFyZHdhcmUgZmVhdHVyZXMgZGlzY292ZXJlZAphdCBib290LCBhbmQgdGhlIHJ1bnRpbWUgc3Rh dGUgb2YgdGhlIHRhc2suICBUbyBtYWtlIHRoZXNlCmRlY2lzaW9ucyBhcyBmYXN0IGFzIHBvc3Np YmxlLCBjb25zdCBjcHVjYXBzIGFyZSB1c2VkIHdoZXJlCmZlYXNpYmxlLCB2aWEgdGhlIHN5c3Rl bV9zdXBwb3J0c19zdmUoKSBoZWxwZXIuCgpUaGUgU1ZFIHJlZ2lzdGVycyBhcmUgb25seSB0cmFj a2VkIGZvciB0aHJlYWRzIHRoYXQgaGF2ZSBleHBsaWNpdGx5CnVzZWQgU1ZFLCBpbmRpY2F0ZWQg YnkgdGhlIG5ldyB0aHJlYWQgZmxhZyBUSUZfU1ZFLiAgT3RoZXJ3aXNlLCB0aGUKRlBTSU1EIHZp ZXcgb2YgdGhlIGFyY2hpdGVjdHVyYWwgc3RhdGUgaXMgc3RvcmVkIGluCnRocmVhZC5mcHNpbWRf c3RhdGUgYXMgdXN1YWwuCgpXaGVuIGluIHVzZSwgdGhlIFNWRSByZWdpc3RlcnMgYXJlIG5vdCBz dG9yZWQgZGlyZWN0bHkgaW4KdGhyZWFkX3N0cnVjdCBkdWUgdG8gdGhlaXIgcG90ZW50aWFsbHkg bGFyZ2UgYW5kIHZhcmlhYmxlIHNpemUuCkJlY2F1c2UgdGhlIHRhc2tfc3RydWN0IHNsYWIgYWxs b2NhdG9yIG11c3QgYmUgY29uZmlndXJlZCB2ZXJ5CmVhcmx5IGR1cmluZyBrZXJuZWwgYm9vdCwg aXQgaXMgYWxzbyB0cmlja3kgdG8gY29uZmlndXJlIGl0CmNvcnJlY3RseSB0byBtYXRjaCB0aGUg bWF4aW11bSB2ZWN0b3IgbGVuZ3RoIHByb3ZpZGVkIGJ5IHRoZQpoYXJkd2FyZSwgc2luY2UgdGhp cyBkZXBlbmRzIG9uIGV4YW1pbmluZyBzZWNvbmRhcnkgQ1BVcyBhcyB3ZWxsIGFzCnRoZSBwcmlt YXJ5LiAgSW5zdGVhZCwgYSBwb2ludGVyIHN2ZV9zdGF0ZSBpbiB0aHJlYWRfc3RydWN0IHBvaW50 cwp0byBhIGR5bmFtaWNhbGx5IGFsbG9jYXRlZCBidWZmZXIgY29udGFpbmluZyB0aGUgU1ZFIHJl Z2lzdGVyIGRhdGEsCmFuZCBjb2RlIGlzIGFkZGVkIHRvIGFsbG9jYXRlIGFuZCBmcmVlIHRoaXMg YnVmZmVyIGF0IGFwcHJvcHJpYXRlCnRpbWVzLgoKVElGX1NWRSBpcyBzZXQgd2hlbiB0YWtpbmcg YW4gU1ZFIGFjY2VzcyB0cmFwIGZyb20gdXNlcnNwYWNlLCBpZgpzdWl0YWJsZSBoYXJkd2FyZSBz dXBwb3J0IGhhcyBiZWVuIGRldGVjdGVkLiAgVGhpcyBlbmFibGVzIFNWRSBmb3IKdGhlIHRocmVh ZDogYSBzdWJzZXF1ZW50IHJldHVybiB0byB1c2Vyc3BhY2Ugd2lsbCBkaXNhYmxlIHRoZSB0cmFw CmFjY29yZGluZ2x5LiAgSWYgc3VjaCBhIHRyYXAgaXMgdGFrZW4gd2l0aG91dCBzdWZmaWNpZW50 IHN5c3RlbS0Kd2lkZSBoYXJkd2FyZSBzdXBwb3J0LCBTSUdJTEwgaXMgc2VudCB0byB0aGUgdGhy ZWFkIGluc3RlYWQgYXMgaWYKYW4gdW5kZWZpbmVkIGluc3RydWN0aW9uIGhhZCBiZWVuIGV4ZWN1 dGVkOiB0aGlzIG1heSBoYXBwZW4gaWYKdXNlcnNwYWNlIHRyaWVzIHRvIHVzZSBTVkUgaW4gYSBz eXN0ZW0gd2hlcmUgbm90IGFsbCBDUFVzIHN1cHBvcnQKaXQgZm9yIGV4YW1wbGUuCgpUaGUga2Vy bmVsIHdpbGwgY2xlYXIgVElGX1NWRSBhbmQgZGlzYWJsZSBTVkUgZm9yIHRoZSB0aHJlYWQKd2hl bmV2ZXIgYW4gZXhwbGljaXQgc3lzY2FsbCBpcyBtYWRlIGJ5IHVzZXJzcGFjZS4gIEZvciBiYWNr d2FyZHMKY29tcGF0aWJpbGl0eSByZWFzb25zIGFuZCBjb25mb3JtYW5jZSB3aXRoIHRoZSBzcGly aXQgb2YgdGhlIGJhc2UKQUFyY2g2NCBwcm9jZWR1cmUgY2FsbCBzdGFuZGFyZCwgdGhlIHN1YnNl dCBvZiB0aGUgU1ZFIHJlZ2lzdGVyCnN0YXRlIHRoYXQgYWxpYXNlcyB0aGUgRlBTSU1EIHJlZ2lz dGVycyBpcyBzdGlsbCBwcmVzZXJ2ZWQgYWNyb3NzIGEKc3lzY2FsbCBldmVuIGlmIHRoaXMgaGFw cGVucy4gIFRoZSByZW1haW5kZXIgb2YgdGhlIFNWRSByZWdpc3RlcgpzdGF0ZSBsb2dpY2FsbHkg YmVjb21lcyB6ZXJvIGF0IHN5c2NhbGwgZW50cnksIHRob3VnaCB0aGUgYWN0dWFsCnplcm9pbmcg d29yayBpcyBjdXJyZW50bHkgZGVmZXJyZWQgdW50aWwgdGhlIHRocmVhZCBuZXh0IHRyaWVzIHRv CnVzZSBTVkUsIGNhdXNpbmcgYW5vdGhlciB0cmFwIHRvIHRoZSBrZXJuZWwuICBUaGlzIGltcGxl bWVudGF0aW9uCmlzIHN1Ym9wdGltYWw6IGluIHRoZSBmdXR1cmUsIHRoZSBmYXN0cGF0aCBjYXNl IG1heSBiZSBvcHRpbWlzZWQKdG8gemVybyB0aGUgcmVnaXN0ZXJzIGluLXBsYWNlIGFuZCBsZWF2 ZSBTVkUgZW5hYmxlZCBmb3IgdGhlIHRhc2ssCndoZXJlIGJlbmVmaWNpYWwuCgpUSUZfU1ZFIGlz IGFsc28gY2xlYXJlZCBpbiB0aGUgZm9sbG93aW5nIHNsb3dwYXRoIGNhc2VzLCB3aGljaCBhcmUK dGFrZW4gYXMgcmVhc29uYWJsZSBoaW50cyB0aGF0IHRoZSB0YXNrIG1heSBubyBsb25nZXIgdXNl IFNWRToKICogZXhlYwogKiBmb3JrIGFuZCBjbG9uZQoKQ29kZSBpcyBhZGRlZCB0byBzeW5jIGRh dGEgYmV0d2VlbiB0aHJlYWQuZnBzaW1kX3N0YXRlIGFuZAp0aHJlYWQuc3ZlX3N0YXRlIHdoZW5l dmVyIGVuYWJsaW5nL2Rpc2FibGluZyBTVkUsIGluIGEgbWFubmVyCmNvbnNpc3RlbnQgd2l0aCB0 aGUgU1ZFIGFyY2hpdGVjdHVyYWwgcHJvZ3JhbW1lcidzIG1vZGVsLgoKU2lnbmVkLW9mZi1ieTog RGF2ZSBNYXJ0aW4gPERhdmUuTWFydGluQGFybS5jb20+ClJldmlld2VkLWJ5OiBDYXRhbGluIE1h cmluYXMgPGNhdGFsaW4ubWFyaW5hc0Bhcm0uY29tPgpDYzogQXJkIEJpZXNoZXV2ZWwgPGFyZC5i aWVzaGV1dmVsQGxpbmFyby5vcmc+CkNjOiBBbGV4IEJlbm7DqWUgPGFsZXguYmVubmVlQGxpbmFy by5vcmc+CgotLS0KCktlcHQgQ2F0YWxpbidzIFJldmlld2VkLWJ5LCBzaW5jZSB0aGlzIGlzIGEg dHJpdmlhbCBjaGFuZ2UuCgpDaGFuZ2VzIHNpbmNlIHY0Ci0tLS0tLS0tLS0tLS0tLS0KCk1pc2Nl bGxhbmVvdXM6CgogKiBNYXJrIGRvX3N2ZV9hY2MoKSBhcyBhc21saW5rYWdlLgoKICAgKFNlbWFu dGljIGNvcnJlY3RuZXNzIG9ubHk7IG5vIGZ1bmN0aW9uYWwgaW1wYWN0LikKLS0tCiBhcmNoL2Fy bTY0L2luY2x1ZGUvYXNtL2Zwc2ltZC5oICAgICAgfCAgMTYgKysKIGFyY2gvYXJtNjQvaW5jbHVk ZS9hc20vcHJvY2Vzc29yLmggICB8ICAgMiArCiBhcmNoL2FybTY0L2luY2x1ZGUvYXNtL3RocmVh ZF9pbmZvLmggfCAgIDQgKwogYXJjaC9hcm02NC9pbmNsdWRlL2FzbS90cmFwcy5oICAgICAgIHwg ICAyICsKIGFyY2gvYXJtNjQva2VybmVsL2VudHJ5LlMgICAgICAgICAgICB8ICAzOSArKysrLQog YXJjaC9hcm02NC9rZXJuZWwvZnBzaW1kLmMgICAgICAgICAgIHwgMzI0ICsrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKystCiBhcmNoL2FybTY0L2tlcm5lbC9wcm9jZXNzLmMgICAgICAg ICAgfCAgMjQgKysrCiBhcmNoL2FybTY0L2tlcm5lbC90cmFwcy5jICAgICAgICAgICAgfCAgIDYg Ky0KIDggZmlsZXMgY2hhbmdlZCwgNDA2IGluc2VydGlvbnMoKyksIDExIGRlbGV0aW9ucygtKQoK ZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vZnBzaW1kLmggYi9hcmNoL2FybTY0 L2luY2x1ZGUvYXNtL2Zwc2ltZC5oCmluZGV4IDAyNmE3YzcuLjU2NTVmZTEgMTAwNjQ0Ci0tLSBh L2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vZnBzaW1kLmgKKysrIGIvYXJjaC9hcm02NC9pbmNsdWRl L2FzbS9mcHNpbWQuaApAQCAtMjAsNiArMjAsOCBAQAogCiAjaWZuZGVmIF9fQVNTRU1CTFlfXwog CisjaW5jbHVkZSA8bGludXgvc3RkZGVmLmg+CisKIC8qCiAgKiBGUC9TSU1EIHN0b3JhZ2UgYXJl YSBoYXM6CiAgKiAgLSBGUFNSIGFuZCBGUENSCkBAIC03Miw2ICs3NCwyMCBAQCBleHRlcm4gdm9p ZCBzdmVfbG9hZF9zdGF0ZSh2b2lkIGNvbnN0ICpzdGF0ZSwgdTMyIGNvbnN0ICpwZnBzciwKIAkJ CSAgIHVuc2lnbmVkIGxvbmcgdnFfbWludXNfMSk7CiBleHRlcm4gdW5zaWduZWQgaW50IHN2ZV9n ZXRfdmwodm9pZCk7CiAKKyNpZmRlZiBDT05GSUdfQVJNNjRfU1ZFCisKK2V4dGVybiBzaXplX3Qg c3ZlX3N0YXRlX3NpemUoc3RydWN0IHRhc2tfc3RydWN0IGNvbnN0ICp0YXNrKTsKKworZXh0ZXJu IHZvaWQgc3ZlX2FsbG9jKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzayk7CitleHRlcm4gdm9pZCBm cHNpbWRfcmVsZWFzZV90YXNrKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzayk7CisKKyNlbHNlIC8q ICEgQ09ORklHX0FSTTY0X1NWRSAqLworCitzdGF0aWMgaW5saW5lIHZvaWQgc3ZlX2FsbG9jKHN0 cnVjdCB0YXNrX3N0cnVjdCAqdGFzaykgeyB9CitzdGF0aWMgaW5saW5lIHZvaWQgZnBzaW1kX3Jl bGVhc2VfdGFzayhzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRhc2spIHsgfQorCisjZW5kaWYgLyogISBD T05GSUdfQVJNNjRfU1ZFICovCisKIC8qIEZvciB1c2UgYnkgRUZJIHJ1bnRpbWUgc2VydmljZXMg Y2FsbHMgb25seSAqLwogZXh0ZXJuIHZvaWQgX19lZmlfZnBzaW1kX2JlZ2luKHZvaWQpOwogZXh0 ZXJuIHZvaWQgX19lZmlfZnBzaW1kX2VuZCh2b2lkKTsKZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQv aW5jbHVkZS9hc20vcHJvY2Vzc29yLmggYi9hcmNoL2FybTY0L2luY2x1ZGUvYXNtL3Byb2Nlc3Nv ci5oCmluZGV4IDdkZGRjYTIuLmUyZjU3NWQgMTAwNjQ0Ci0tLSBhL2FyY2gvYXJtNjQvaW5jbHVk ZS9hc20vcHJvY2Vzc29yLmgKKysrIGIvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS9wcm9jZXNzb3Iu aApAQCAtMTA1LDYgKzEwNSw4IEBAIHN0cnVjdCB0aHJlYWRfc3RydWN0IHsKIAl1bnNpZ25lZCBs b25nCQl0cDJfdmFsdWU7CiAjZW5kaWYKIAlzdHJ1Y3QgZnBzaW1kX3N0YXRlCWZwc2ltZF9zdGF0 ZTsKKwl2b2lkCQkJKnN2ZV9zdGF0ZTsJLyogU1ZFIHJlZ2lzdGVycywgaWYgYW55ICovCisJdW5z aWduZWQgaW50CQlzdmVfdmw7CQkvKiBTVkUgdmVjdG9yIGxlbmd0aCAqLwogCXVuc2lnbmVkIGxv bmcJCWZhdWx0X2FkZHJlc3M7CS8qIGZhdWx0IGluZm8gKi8KIAl1bnNpZ25lZCBsb25nCQlmYXVs dF9jb2RlOwkvKiBFU1JfRUwxIHZhbHVlICovCiAJc3RydWN0IGRlYnVnX2luZm8JZGVidWc7CQkv KiBkZWJ1Z2dpbmcgKi8KZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vdGhyZWFk X2luZm8uaCBiL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20vdGhyZWFkX2luZm8uaAppbmRleCBkZGRl ZDY0Li45MmI3YjQ4IDEwMDY0NAotLS0gYS9hcmNoL2FybTY0L2luY2x1ZGUvYXNtL3RocmVhZF9p bmZvLmgKKysrIGIvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS90aHJlYWRfaW5mby5oCkBAIC02Myw2 ICs2Myw4IEBAIHN0cnVjdCB0aHJlYWRfaW5mbyB7CiB2b2lkIGFyY2hfc2V0dXBfbmV3X2V4ZWMo dm9pZCk7CiAjZGVmaW5lIGFyY2hfc2V0dXBfbmV3X2V4ZWMgICAgIGFyY2hfc2V0dXBfbmV3X2V4 ZWMKIAordm9pZCBhcmNoX3JlbGVhc2VfdGFza19zdHJ1Y3Qoc3RydWN0IHRhc2tfc3RydWN0ICp0 c2spOworCiAjZW5kaWYKIAogLyoKQEAgLTkyLDYgKzk0LDcgQEAgdm9pZCBhcmNoX3NldHVwX25l d19leGVjKHZvaWQpOwogI2RlZmluZSBUSUZfUkVTVE9SRV9TSUdNQVNLCTIwCiAjZGVmaW5lIFRJ Rl9TSU5HTEVTVEVQCQkyMQogI2RlZmluZSBUSUZfMzJCSVQJCTIyCS8qIDMyYml0IHByb2Nlc3Mg Ki8KKyNkZWZpbmUgVElGX1NWRQkJCTIzCS8qIFNjYWxhYmxlIFZlY3RvciBFeHRlbnNpb24gaW4g dXNlICovCiAKICNkZWZpbmUgX1RJRl9TSUdQRU5ESU5HCQkoMSA8PCBUSUZfU0lHUEVORElORykK ICNkZWZpbmUgX1RJRl9ORUVEX1JFU0NIRUQJKDEgPDwgVElGX05FRURfUkVTQ0hFRCkKQEAgLTEw NSw2ICsxMDgsNyBAQCB2b2lkIGFyY2hfc2V0dXBfbmV3X2V4ZWModm9pZCk7CiAjZGVmaW5lIF9U SUZfVVBST0JFCQkoMSA8PCBUSUZfVVBST0JFKQogI2RlZmluZSBfVElGX0ZTQ0hFQ0sJCSgxIDw8 IFRJRl9GU0NIRUNLKQogI2RlZmluZSBfVElGXzMyQklUCQkoMSA8PCBUSUZfMzJCSVQpCisjZGVm aW5lIF9USUZfU1ZFCQkoMSA8PCBUSUZfU1ZFKQogCiAjZGVmaW5lIF9USUZfV09SS19NQVNLCQko X1RJRl9ORUVEX1JFU0NIRUQgfCBfVElGX1NJR1BFTkRJTkcgfCBcCiAJCQkJIF9USUZfTk9USUZZ X1JFU1VNRSB8IF9USUZfRk9SRUlHTl9GUFNUQVRFIHwgXApkaWZmIC0tZ2l0IGEvYXJjaC9hcm02 NC9pbmNsdWRlL2FzbS90cmFwcy5oIGIvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS90cmFwcy5oCmlu ZGV4IDQ1ZTNkYTMuLjE2OTZmOWQgMTAwNjQ0Ci0tLSBhL2FyY2gvYXJtNjQvaW5jbHVkZS9hc20v dHJhcHMuaAorKysgYi9hcmNoL2FybTY0L2luY2x1ZGUvYXNtL3RyYXBzLmgKQEAgLTM0LDYgKzM0 LDggQEAgc3RydWN0IHVuZGVmX2hvb2sgewogCiB2b2lkIHJlZ2lzdGVyX3VuZGVmX2hvb2soc3Ry dWN0IHVuZGVmX2hvb2sgKmhvb2spOwogdm9pZCB1bnJlZ2lzdGVyX3VuZGVmX2hvb2soc3RydWN0 IHVuZGVmX2hvb2sgKmhvb2spOwordm9pZCBmb3JjZV9zaWduYWxfaW5qZWN0KGludCBzaWduYWws IGludCBjb2RlLCBzdHJ1Y3QgcHRfcmVncyAqcmVncywKKwkJCSB1bnNpZ25lZCBsb25nIGFkZHJl c3MpOwogCiB2b2lkIGFybTY0X25vdGlmeV9zZWdmYXVsdChzdHJ1Y3QgcHRfcmVncyAqcmVncywg dW5zaWduZWQgbG9uZyBhZGRyKTsKIApkaWZmIC0tZ2l0IGEvYXJjaC9hcm02NC9rZXJuZWwvZW50 cnkuUyBiL2FyY2gvYXJtNjQva2VybmVsL2VudHJ5LlMKaW5kZXggZjVlODUxZS4uNTZlODQ4ZiAx MDA2NDQKLS0tIGEvYXJjaC9hcm02NC9rZXJuZWwvZW50cnkuUworKysgYi9hcmNoL2FybTY0L2tl cm5lbC9lbnRyeS5TCkBAIC02MDcsNiArNjA3LDggQEAgZWwwX3N5bmM6CiAJYi5lcQllbDBfaWEK IAljbXAJeDI0LCAjRVNSX0VMeF9FQ19GUF9BU0lNRAkvLyBGUC9BU0lNRCBhY2Nlc3MKIAliLmVx CWVsMF9mcHNpbWRfYWNjCisJY21wCXgyNCwgI0VTUl9FTHhfRUNfU1ZFCQkvLyBTVkUgYWNjZXNz CisJYi5lcQllbDBfc3ZlX2FjYwogCWNtcAl4MjQsICNFU1JfRUx4X0VDX0ZQX0VYQzY0CS8vIEZQ L0FTSU1EIGV4Y2VwdGlvbgogCWIuZXEJZWwwX2Zwc2ltZF9leGMKIAljbXAJeDI0LCAjRVNSX0VM eF9FQ19TWVM2NAkJLy8gY29uZmlndXJhYmxlIHRyYXAKQEAgLTY1OCw2ICs2NjAsNyBAQCBlbDBf c3ZjX2NvbXBhdDoKIAkvKgogCSAqIEFBcmNoMzIgc3lzY2FsbCBoYW5kbGluZwogCSAqLworCWxk cgl4MTYsIFt0c2ssICNUU0tfVElfRkxBR1NdCS8vIGxvYWQgdGhyZWFkIGZsYWdzCiAJYWRycAlz dGJsLCBjb21wYXRfc3lzX2NhbGxfdGFibGUJLy8gbG9hZCBjb21wYXQgc3lzY2FsbCB0YWJsZSBw b2ludGVyCiAJbW92CXdzY25vLCB3NwkJCS8vIHN5c2NhbGwgbnVtYmVyIGluIHc3IChyNykKIAlt b3YgICAgIHdzY19uciwgI19fTlJfY29tcGF0X3N5c2NhbGxzCkBAIC03MDUsOSArNzA4LDE5IEBA IGVsMF9mcHNpbWRfYWNjOgogCW1vdgl4MSwgc3AKIAlibAlkb19mcHNpbWRfYWNjCiAJYglyZXRf dG9fdXNlcgorZWwwX3N2ZV9hY2M6CisJLyoKKwkgKiBTY2FsYWJsZSBWZWN0b3IgRXh0ZW5zaW9u IGFjY2VzcworCSAqLworCWVuYWJsZV9kYmdfYW5kX2lycQorCWN0X3VzZXJfZXhpdAorCW1vdgl4 MCwgeDI1CisJbW92CXgxLCBzcAorCWJsCWRvX3N2ZV9hY2MKKwliCXJldF90b191c2VyCiBlbDBf ZnBzaW1kX2V4YzoKIAkvKgotCSAqIEZsb2F0aW5nIFBvaW50IG9yIEFkdmFuY2VkIFNJTUQgZXhj ZXB0aW9uCisJICogRmxvYXRpbmcgUG9pbnQsIEFkdmFuY2VkIFNJTUQgb3IgU1ZFIGV4Y2VwdGlv bgogCSAqLwogCWVuYWJsZV9kYmcKIAljdF91c2VyX2V4aXQKQEAgLTgzNSwxNiArODQ4LDM2IEBA IEVORFBST0MocmV0X3RvX3VzZXIpCiAgKi8KIAkuYWxpZ24JNgogZWwwX3N2YzoKKwlsZHIJeDE2 LCBbdHNrLCAjVFNLX1RJX0ZMQUdTXQkvLyBsb2FkIHRocmVhZCBmbGFncwogCWFkcnAJc3RibCwg c3lzX2NhbGxfdGFibGUJCS8vIGxvYWQgc3lzY2FsbCB0YWJsZSBwb2ludGVyCiAJbW92CXdzY25v LCB3OAkJCS8vIHN5c2NhbGwgbnVtYmVyIGluIHc4CiAJbW92CXdzY19uciwgI19fTlJfc3lzY2Fs bHMKKworI2lmbmRlZiBDT05GSUdfQVJNNjRfU1ZFCisJYgllbDBfc3ZjX25ha2VkCisjZWxzZQor CXRiegl4MTYsICNUSUZfU1ZFLCBlbDBfc3ZjX25ha2VkCS8vIFNraXAgdW5sZXNzIFRJRl9TVkUg c2V0OgorCWJpYwl4MTYsIHgxNiwgI19USUZfU1ZFCQkvLyBkaXNjYXJkIFNWRSBzdGF0ZQorCXN0 cgl4MTYsIFt0c2ssICNUU0tfVElfRkxBR1NdCisKKwkvKgorCSAqIHRhc2tfZnBzaW1kX2xvYWQo KSB3b24ndCBiZSBjYWxsZWQgdG8gdXBkYXRlIENQQUNSX0VMMSBpbgorCSAqIHJldF90b191c2Vy IHVubGVzcyBUSUZfRk9SRUlHTl9GUFNUQVRFIGlzIHN0aWxsIHNldCwgd2hpY2ggb25seQorCSAq IGhhcHBlbnMgaWYgYSBjb250ZXh0IHN3aXRjaCBvciBrZXJuZWxfbmVvbl9iZWdpbigpIG9yIGNv bnRleHQKKwkgKiBtb2RpZmljYXRpb24gKHNpZ3JldHVybiwgcHRyYWNlKSBpbnRlcnZlbmVzLgor CSAqIFNvLCBlbnN1cmUgdGhhdCBDUEFDUl9FTDEgaXMgYWxyZWFkeSBjb3JyZWN0IGZvciB0aGUg ZmFzdC1wYXRoIGNhc2U6CisJICovCisJbXJzCXg5LCBjcGFjcl9lbDEKKwliaWMJeDksIHg5LCAj Q1BBQ1JfRUwxX1pFTl9FTDBFTgkvLyBkaXNhYmxlIFNWRSBmb3IgZWwwCisJbXNyCWNwYWNyX2Vs MSwgeDkJCQkvLyBzeW5jaHJvbmlzZWQgYnkgZXJldCB0byBlbDAKKyNlbmRpZiAvKiBDT05GSUdf QVJNNjRfU1ZFICovCisKIGVsMF9zdmNfbmFrZWQ6CQkJCQkvLyBjb21wYXQgZW50cnkgcG9pbnQK IAlzdHAJeDAsIHhzY25vLCBbc3AsICNTX09SSUdfWDBdCS8vIHNhdmUgdGhlIG9yaWdpbmFsIHgw IGFuZCBzeXNjYWxsIG51bWJlcgogCWVuYWJsZV9kYmdfYW5kX2lycQogCWN0X3VzZXJfZXhpdCAx CiAKLQlsZHIJeDE2LCBbdHNrLCAjVFNLX1RJX0ZMQUdTXQkvLyBjaGVjayBmb3Igc3lzY2FsbCBo b29rcwotCXRzdAl4MTYsICNfVElGX1NZU0NBTExfV09SSworCXRzdAl4MTYsICNfVElGX1NZU0NB TExfV09SSwkJLy8gY2hlY2sgZm9yIHN5c2NhbGwgaG9va3MKIAliLm5lCV9fc3lzX3RyYWNlCiAJ Y21wICAgICB3c2Nubywgd3NjX25yCQkJLy8gY2hlY2sgdXBwZXIgc3lzY2FsbCBsaW1pdAogCWIu aHMJbmlfc3lzCmRpZmYgLS1naXQgYS9hcmNoL2FybTY0L2tlcm5lbC9mcHNpbWQuYyBiL2FyY2gv YXJtNjQva2VybmVsL2Zwc2ltZC5jCmluZGV4IDJiYWJhMGQuLjc4N2Y1ZDMgMTAwNjQ0Ci0tLSBh L2FyY2gvYXJtNjQva2VybmVsL2Zwc2ltZC5jCisrKyBiL2FyY2gvYXJtNjQva2VybmVsL2Zwc2lt ZC5jCkBAIC0xOCwxOSArMTgsMjcgQEAKICAqLwogCiAjaW5jbHVkZSA8bGludXgvYm90dG9tX2hh bGYuaD4KKyNpbmNsdWRlIDxsaW51eC9idWcuaD4KKyNpbmNsdWRlIDxsaW51eC9jb21wYXQuaD4K ICNpbmNsdWRlIDxsaW51eC9jcHUuaD4KICNpbmNsdWRlIDxsaW51eC9jcHVfcG0uaD4KICNpbmNs dWRlIDxsaW51eC9rZXJuZWwuaD4KICNpbmNsdWRlIDxsaW51eC9saW5rYWdlLmg+CisjaW5jbHVk ZSA8bGludXgvaXJxZmxhZ3MuaD4KICNpbmNsdWRlIDxsaW51eC9pbml0Lmg+CiAjaW5jbHVkZSA8 bGludXgvcGVyY3B1Lmg+CiAjaW5jbHVkZSA8bGludXgvcHJlZW1wdC5oPgorI2luY2x1ZGUgPGxp bnV4L3B0cmFjZS5oPgogI2luY2x1ZGUgPGxpbnV4L3NjaGVkL3NpZ25hbC5oPgogI2luY2x1ZGUg PGxpbnV4L3NpZ25hbC5oPgorI2luY2x1ZGUgPGxpbnV4L3NsYWIuaD4KIAogI2luY2x1ZGUgPGFz bS9mcHNpbWQuaD4KICNpbmNsdWRlIDxhc20vY3B1dHlwZS5oPgogI2luY2x1ZGUgPGFzbS9zaW1k Lmg+CisjaW5jbHVkZSA8YXNtL3NpZ2NvbnRleHQuaD4KKyNpbmNsdWRlIDxhc20vc3lzcmVnLmg+ CisjaW5jbHVkZSA8YXNtL3RyYXBzLmg+CiAKICNkZWZpbmUgRlBFWENfSU9GCSgxIDw8IDApCiAj ZGVmaW5lIEZQRVhDX0RaRgkoMSA8PCAxKQpAQCAtNDAsNiArNDgsOCBAQAogI2RlZmluZSBGUEVY Q19JREYJKDEgPDwgNykKIAogLyoKKyAqIChOb3RlOiBpbiB0aGlzIGRpc2N1c3Npb24sIHN0YXRl bWVudHMgYWJvdXQgRlBTSU1EIGFwcGx5IGVxdWFsbHkgdG8gU1ZFLikKKyAqCiAgKiBJbiBvcmRl ciB0byByZWR1Y2UgdGhlIG51bWJlciBvZiB0aW1lcyB0aGUgRlBTSU1EIHN0YXRlIGlzIG5lZWRs ZXNzbHkgc2F2ZWQKICAqIGFuZCByZXN0b3JlZCwgd2UgbmVlZCB0byBrZWVwIHRyYWNrIG9mIHR3 byB0aGluZ3M6CiAgKiAoYSkgZm9yIGVhY2ggdGFzaywgd2UgbmVlZCB0byByZW1lbWJlciB3aGlj aCBDUFUgd2FzIHRoZSBsYXN0IG9uZSB0byBoYXZlCkBAIC0xMDEsNiArMTExLDI3OSBAQAogc3Rh dGljIERFRklORV9QRVJfQ1BVKHN0cnVjdCBmcHNpbWRfc3RhdGUgKiwgZnBzaW1kX2xhc3Rfc3Rh dGUpOwogCiAvKgorICogQ2FsbCBfX3N2ZV9mcmVlKCkgZGlyZWN0bHkgb25seSBpZiB5b3Uga25v dyB0YXNrIGNhbid0IGJlIHNjaGVkdWxlZAorICogb3IgcHJlZW1wdGVkLgorICovCitzdGF0aWMg dm9pZCBfX3N2ZV9mcmVlKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzaykKK3sKKwlrZnJlZSh0YXNr LT50aHJlYWQuc3ZlX3N0YXRlKTsKKwl0YXNrLT50aHJlYWQuc3ZlX3N0YXRlID0gTlVMTDsKK30K Kworc3RhdGljIHZvaWQgc3ZlX2ZyZWUoc3RydWN0IHRhc2tfc3RydWN0ICp0YXNrKQoreworCVdB Uk5fT04odGVzdF90c2tfdGhyZWFkX2ZsYWcodGFzaywgVElGX1NWRSkpOworCisJX19zdmVfZnJl ZSh0YXNrKTsKK30KKworCisvKiBPZmZzZXQgb2YgRkZSIGluIHRoZSBTVkUgcmVnaXN0ZXIgZHVt cCAqLworc3RhdGljIHNpemVfdCBzdmVfZmZyX29mZnNldChpbnQgdmwpCit7CisJcmV0dXJuIFNW RV9TSUdfRkZSX09GRlNFVChzdmVfdnFfZnJvbV92bCh2bCkpIC0gU1ZFX1NJR19SRUdTX09GRlNF VDsKK30KKworc3RhdGljIHZvaWQgKnN2ZV9wZmZyKHN0cnVjdCB0YXNrX3N0cnVjdCAqdGFzaykK K3sKKwlyZXR1cm4gKGNoYXIgKil0YXNrLT50aHJlYWQuc3ZlX3N0YXRlICsKKwkJc3ZlX2Zmcl9v ZmZzZXQodGFzay0+dGhyZWFkLnN2ZV92bCk7Cit9CisKK3N0YXRpYyB2b2lkIGNoYW5nZV9jcGFj cih1NjQgdmFsLCB1NjQgbWFzaykKK3sKKwl1NjQgY3BhY3IgPSByZWFkX3N5c3JlZyhDUEFDUl9F TDEpOworCXU2NCBuZXcgPSAoY3BhY3IgJiB+bWFzaykgfCB2YWw7CisKKwlpZiAobmV3ICE9IGNw YWNyKQorCQl3cml0ZV9zeXNyZWcobmV3LCBDUEFDUl9FTDEpOworfQorCitzdGF0aWMgdm9pZCBz dmVfdXNlcl9kaXNhYmxlKHZvaWQpCit7CisJY2hhbmdlX2NwYWNyKDAsIENQQUNSX0VMMV9aRU5f RUwwRU4pOworfQorCitzdGF0aWMgdm9pZCBzdmVfdXNlcl9lbmFibGUodm9pZCkKK3sKKwljaGFu Z2VfY3BhY3IoQ1BBQ1JfRUwxX1pFTl9FTDBFTiwgQ1BBQ1JfRUwxX1pFTl9FTDBFTik7Cit9CisK Ky8qCisgKiBUSUZfU1ZFIGNvbnRyb2xzIHdoZXRoZXIgYSB0YXNrIGNhbiB1c2UgU1ZFIHdpdGhv dXQgdHJhcHBpbmcgd2hpbGUKKyAqIGluIHVzZXJzcGFjZSwgYW5kIGFsc28gdGhlIHdheSBhIHRh c2sncyBGUFNJTUQvU1ZFIHN0YXRlIGlzIHN0b3JlZAorICogaW4gdGhyZWFkX3N0cnVjdC4KKyAq CisgKiBUaGUga2VybmVsIHVzZXMgdGhpcyBmbGFnIHRvIHRyYWNrIHdoZXRoZXIgYSB1c2VyIHRh c2sgaXMgYWN0aXZlbHkKKyAqIHVzaW5nIFNWRSwgYW5kIHRoZXJlZm9yZSB3aGV0aGVyIGZ1bGwg U1ZFIHJlZ2lzdGVyIHN0YXRlIG5lZWRzIHRvCisgKiBiZSB0cmFja2VkLiAgSWYgbm90LCB0aGUg Y2hlYXBlciBGUFNJTUQgY29udGV4dCBoYW5kbGluZyBjb2RlIGNhbgorICogYmUgdXNlZCBpbnN0 ZWFkIG9mIHRoZSBtb3JlIGNvc3RseSBTVkUgZXF1aXZhbGVudHMuCisgKgorICogICogVElGX1NW RSBzZXQ6CisgKgorICogICAgVGhlIHRhc2sgY2FuIGV4ZWN1dGUgU1ZFIGluc3RydWN0aW9ucyB3 aGlsZSBpbiB1c2Vyc3BhY2Ugd2l0aG91dAorICogICAgdHJhcHBpbmcgdG8gdGhlIGtlcm5lbC4K KyAqCisgKiAgICBXaGVuIHN0b3JlZCwgWjAtWjMxIChpbmNvcnBvcmF0aW5nIFZuIGluIGJpdHNb MTI3OjBdIG9yIHRoZQorICogICAgY29ycmVzcG9uZGluZyBabiksIFAwLVAxNSBhbmQgRkZSIGFy ZSBlbmNvZGVkIGluIGluCisgKiAgICB0YXNrLT50aHJlYWQuc3ZlX3N0YXRlLCBmb3JtYXR0ZWQg YXBwcm9wcmlhdGVseSBmb3IgdmVjdG9yCisgKiAgICBsZW5ndGggdGFzay0+dGhyZWFkLnN2ZV92 bC4KKyAqCisgKiAgICB0YXNrLT50aHJlYWQuc3ZlX3N0YXRlIG11c3QgcG9pbnQgdG8gYSB2YWxp ZCBidWZmZXIgYXQgbGVhc3QKKyAqICAgIHN2ZV9zdGF0ZV9zaXplKHRhc2spIGJ5dGVzIGluIHNp emUuCisgKgorICogICAgRHVyaW5nIGFueSBzeXNjYWxsLCB0aGUga2VybmVsIG1heSBvcHRpb25h bGx5IGNsZWFyIFRJRl9TVkUgYW5kCisgKiAgICBkaXNjYXJkIHRoZSB2ZWN0b3Igc3RhdGUgZXhj ZXB0IGZvciB0aGUgRlBTSU1EIHN1YnNldC4KKyAqCisgKiAgKiBUSUZfU1ZFIGNsZWFyOgorICoK KyAqICAgIEFuIGF0dGVtcHQgYnkgdGhlIHVzZXIgdGFzayB0byBleGVjdXRlIGFuIFNWRSBpbnN0 cnVjdGlvbiBjYXVzZXMKKyAqICAgIGRvX3N2ZV9hY2MoKSB0byBiZSBjYWxsZWQsIHdoaWNoIGRv ZXMgc29tZSBwcmVwYXJhdGlvbiBhbmQgdGhlbgorICogICAgc2V0cyBUSUZfU1ZFLgorICoKKyAq ICAgIFdoZW4gc3RvcmVkLCBGUFNJTUQgcmVnaXN0ZXJzIFYwLVYzMSBhcmUgZW5jb2RlZCBpbgor ICogICAgdGFzay0+ZnBzaW1kX3N0YXRlOyBiaXRzIFttYXggOiAxMjhdIGZvciBlYWNoIG9mIFow LVozMSBhcmUKKyAqICAgIGxvZ2ljYWxseSB6ZXJvIGJ1dCBub3Qgc3RvcmVkIGFueXdoZXJlOyBQ MC1QMTUgYW5kIEZGUiBhcmUgbm90CisgKiAgICBzdG9yZWQgYW5kIGhhdmUgdW5zcGVjaWZpZWQg dmFsdWVzIGZyb20gdXNlcnNwYWNlJ3MgcG9pbnQgb2YKKyAqICAgIHZpZXcuICBGb3IgaHlnaWVu ZSBwdXJwb3NlcywgdGhlIGtlcm5lbCB6ZXJvZXMgdGhlbSBvbiBuZXh0IHVzZSwKKyAqICAgIGJ1 dCB1c2Vyc3BhY2UgaXMgZGlzY291cmFnZWQgZnJvbSByZWx5aW5nIG9uIHRoaXMuCisgKgorICog ICAgdGFzay0+dGhyZWFkLnN2ZV9zdGF0ZSBkb2VzIG5vdCBuZWVkIHRvIGJlIG5vbi1OVUxMLCB2 YWxpZCBvciBhbnkKKyAqICAgIHBhcnRpY3VsYXIgc2l6ZTogaXQgbXVzdCBub3QgYmUgZGVyZWZl cmVuY2VkLgorICoKKyAqICAqIEZQU1IgYW5kIEZQQ1IgYXJlIGFsd2F5cyBzdG9yZWQgaW4gdGFz ay0+ZnBzaW1kX3N0YXRlIGlycmVzcGN0aXZlIG9mCisgKiAgICB3aGV0aGVyIFRJRl9TVkUgaXMg Y2xlYXIgb3Igc2V0LCBzaW5jZSB0aGVzZSBhcmUgbm90IHZlY3RvciBsZW5ndGgKKyAqICAgIGRl cGVuZGVudC4KKyAqLworCisvKgorICogVXBkYXRlIGN1cnJlbnQncyBGUFNJTUQvU1ZFIHJlZ2lz dGVycyBmcm9tIHRocmVhZF9zdHJ1Y3QuCisgKgorICogVGhpcyBmdW5jdGlvbiBzaG91bGQgYmUg Y2FsbGVkIG9ubHkgd2hlbiB0aGUgRlBTSU1EL1NWRSBzdGF0ZSBpbgorICogdGhyZWFkX3N0cnVj dCBpcyBrbm93biB0byBiZSB1cCB0byBkYXRlLCB3aGVuIHByZXBhcmluZyB0byBlbnRlcgorICog dXNlcnNwYWNlLgorICoKKyAqIFNvZnRpcnFzIChhbmQgcHJlZW1wdGlvbikgbXVzdCBiZSBkaXNh YmxlZC4KKyAqLworc3RhdGljIHZvaWQgdGFza19mcHNpbWRfbG9hZCh2b2lkKQoreworCVdBUk5f T04oIWluX3NvZnRpcnEoKSAmJiAhaXJxc19kaXNhYmxlZCgpKTsKKworCWlmIChzeXN0ZW1fc3Vw cG9ydHNfc3ZlKCkgJiYgdGVzdF90aHJlYWRfZmxhZyhUSUZfU1ZFKSkKKwkJc3ZlX2xvYWRfc3Rh dGUoc3ZlX3BmZnIoY3VycmVudCksCisJCQkgICAgICAgJmN1cnJlbnQtPnRocmVhZC5mcHNpbWRf c3RhdGUuZnBzciwKKwkJCSAgICAgICBzdmVfdnFfZnJvbV92bChjdXJyZW50LT50aHJlYWQuc3Zl X3ZsKSAtIDEpOworCWVsc2UKKwkJZnBzaW1kX2xvYWRfc3RhdGUoJmN1cnJlbnQtPnRocmVhZC5m cHNpbWRfc3RhdGUpOworCisJaWYgKHN5c3RlbV9zdXBwb3J0c19zdmUoKSkgeworCQkvKiBUb2dn bGUgU1ZFIHRyYXBwaW5nIGZvciB1c2Vyc3BhY2UgaWYgbmVlZGVkICovCisJCWlmICh0ZXN0X3Ro cmVhZF9mbGFnKFRJRl9TVkUpKQorCQkJc3ZlX3VzZXJfZW5hYmxlKCk7CisJCWVsc2UKKwkJCXN2 ZV91c2VyX2Rpc2FibGUoKTsKKworCQkvKiBTZXJpYWxpc2VkIGJ5IGV4Y2VwdGlvbiByZXR1cm4g dG8gdXNlciAqLworCX0KK30KKworLyoKKyAqIEVuc3VyZSBjdXJyZW50J3MgRlBTSU1EL1NWRSBz dG9yYWdlIGluIHRocmVhZF9zdHJ1Y3QgaXMgdXAgdG8gZGF0ZQorICogd2l0aCByZXNwZWN0IHRv IHRoZSBDUFUgcmVnaXN0ZXJzLgorICoKKyAqIFNvZnRpcnFzIChhbmQgcHJlZW1wdGlvbikgbXVz dCBiZSBkaXNhYmxlZC4KKyAqLworc3RhdGljIHZvaWQgdGFza19mcHNpbWRfc2F2ZSh2b2lkKQor eworCVdBUk5fT04oIWluX3NvZnRpcnEoKSAmJiAhaXJxc19kaXNhYmxlZCgpKTsKKworCWlmICgh dGVzdF90aHJlYWRfZmxhZyhUSUZfRk9SRUlHTl9GUFNUQVRFKSkgeworCQlpZiAoc3lzdGVtX3N1 cHBvcnRzX3N2ZSgpICYmIHRlc3RfdGhyZWFkX2ZsYWcoVElGX1NWRSkpIHsKKwkJCWlmIChXQVJO X09OKHN2ZV9nZXRfdmwoKSAhPSBjdXJyZW50LT50aHJlYWQuc3ZlX3ZsKSkgeworCQkJCS8qCisJ CQkJICogQ2FuJ3Qgc2F2ZSB0aGUgdXNlciByZWdzLCBzbyBjdXJyZW50IHdvdWxkCisJCQkJICog cmUtZW50ZXIgdXNlciB3aXRoIGNvcnJ1cHQgc3RhdGUuCisJCQkJICogVGhlcmUncyBubyB3YXkg dG8gcmVjb3Zlciwgc28ga2lsbCBpdDoKKwkJCQkgKi8KKwkJCQlmb3JjZV9zaWduYWxfaW5qZWN0 KAorCQkJCQlTSUdLSUxMLCAwLCBjdXJyZW50X3B0X3JlZ3MoKSwgMCk7CisJCQkJcmV0dXJuOwor CQkJfQorCisJCQlzdmVfc2F2ZV9zdGF0ZShzdmVfcGZmcihjdXJyZW50KSwKKwkJCQkgICAgICAg JmN1cnJlbnQtPnRocmVhZC5mcHNpbWRfc3RhdGUuZnBzcik7CisJCX0gZWxzZQorCQkJZnBzaW1k X3NhdmVfc3RhdGUoJmN1cnJlbnQtPnRocmVhZC5mcHNpbWRfc3RhdGUpOworCX0KK30KKworI2Rl ZmluZSBaUkVHKHN2ZV9zdGF0ZSwgdnEsIG4pICgoY2hhciAqKShzdmVfc3RhdGUpICsJCVwKKwko U1ZFX1NJR19aUkVHX09GRlNFVCh2cSwgbikgLSBTVkVfU0lHX1JFR1NfT0ZGU0VUKSkKKworLyoK KyAqIFRyYW5zZmVyIHRoZSBGUFNJTUQgc3RhdGUgaW4gdGFzay0+dGhyZWFkLmZwc2ltZF9zdGF0 ZSB0bworICogdGFzay0+dGhyZWFkLnN2ZV9zdGF0ZS4KKyAqCisgKiBUYXNrIGNhbiBiZSBhIG5v bi1ydW5uYWJsZSB0YXNrLCBvciBjdXJyZW50LiAgSW4gdGhlIGxhdHRlciBjYXNlLAorICogc29m dGlycXMgKGFuZCBwcmVlbXB0aW9uKSBtdXN0IGJlIGRpc2FibGVkLgorICogdGFzay0+dGhyZWFk LnN2ZV9zdGF0ZSBtdXN0IHBvaW50IHRvIGF0IGxlYXN0IHN2ZV9zdGF0ZV9zaXplKHRhc2spCisg KiBieXRlcyBvZiBhbGxvY2F0ZWQga2VybmVsIG1lbW9yeS4KKyAqIHRhc2stPnRocmVhZC5mcHNp bWRfc3RhdGUgbXVzdCBiZSB1cCB0byBkYXRlIGJlZm9yZSBjYWxsaW5nIHRoaXMgZnVuY3Rpb24u CisgKi8KK3N0YXRpYyB2b2lkIGZwc2ltZF90b19zdmUoc3RydWN0IHRhc2tfc3RydWN0ICp0YXNr KQoreworCXVuc2lnbmVkIGludCB2cTsKKwl2b2lkICpzc3QgPSB0YXNrLT50aHJlYWQuc3ZlX3N0 YXRlOworCXN0cnVjdCBmcHNpbWRfc3RhdGUgY29uc3QgKmZzdCA9ICZ0YXNrLT50aHJlYWQuZnBz aW1kX3N0YXRlOworCXVuc2lnbmVkIGludCBpOworCisJaWYgKCFzeXN0ZW1fc3VwcG9ydHNfc3Zl KCkpCisJCXJldHVybjsKKworCXZxID0gc3ZlX3ZxX2Zyb21fdmwodGFzay0+dGhyZWFkLnN2ZV92 bCk7CisJZm9yIChpID0gMDsgaSA8IDMyOyArK2kpCisJCW1lbWNweShaUkVHKHNzdCwgdnEsIGkp LCAmZnN0LT52cmVnc1tpXSwKKwkJICAgICAgIHNpemVvZihmc3QtPnZyZWdzW2ldKSk7Cit9CisK KyNpZmRlZiBDT05GSUdfQVJNNjRfU1ZFCisKKy8qCisgKiBSZXR1cm4gaG93IG1hbnkgYnl0ZXMg b2YgbWVtb3J5IGFyZSByZXF1aXJlZCB0byBzdG9yZSB0aGUgZnVsbCBTVkUKKyAqIHN0YXRlIGZv ciB0YXNrLCBnaXZlbiB0YXNrJ3MgY3VycmVudGx5IGNvbmZpZ3VyZWQgdmVjdG9yIGxlbmd0aC4K KyAqLworc2l6ZV90IHN2ZV9zdGF0ZV9zaXplKHN0cnVjdCB0YXNrX3N0cnVjdCBjb25zdCAqdGFz aykKK3sKKwlyZXR1cm4gU1ZFX1NJR19SRUdTX1NJWkUoc3ZlX3ZxX2Zyb21fdmwodGFzay0+dGhy ZWFkLnN2ZV92bCkpOworfQorCisvKgorICogRW5zdXJlIHRoYXQgdGFzay0+dGhyZWFkLnN2ZV9z dGF0ZSBpcyBhbGxvY2F0ZWQgYW5kIHN1ZmZpY2llbnRseSBsYXJnZS4KKyAqCisgKiBUaGlzIGZ1 bmN0aW9uIHNob3VsZCBiZSB1c2VkIG9ubHkgaW4gcHJlcGFyYXRpb24gZm9yIHJlcGxhY2luZwor ICogdGFzay0+dGhyZWFkLnN2ZV9zdGF0ZSB3aXRoIG5ldyBkYXRhLiAgVGhlIG1lbW9yeSBpcyBh bHdheXMgemVyb2VkCisgKiBoZXJlIHRvIHByZXZlbnQgc3RhbGUgZGF0YSBmcm9tIHNob3dpbmcg dGhyb3VnaDogdGhpcyBpcyBkb25lIGluCisgKiB0aGUgaW50ZXJlc3Qgb2YgdGVzdGFiaWxpdHkg YW5kIHByZWRpY3RhYmlsaXR5OiBleGNlcHQgaW4gdGhlCisgKiBkb19zdmVfYWNjKCkgY2FzZSwg dGhlcmUgaXMgbm8gQUJJIHJlcXVpcmVtZW50IHRvIGhpZGUgc3RhbGUgZGF0YQorICogd3JpdHRl biBwcmV2aW91c2x5IGJlIHRhc2suCisgKi8KK3ZvaWQgc3ZlX2FsbG9jKHN0cnVjdCB0YXNrX3N0 cnVjdCAqdGFzaykKK3sKKwlpZiAodGFzay0+dGhyZWFkLnN2ZV9zdGF0ZSkgeworCQltZW1zZXQo dGFzay0+dGhyZWFkLnN2ZV9zdGF0ZSwgMCwgc3ZlX3N0YXRlX3NpemUoY3VycmVudCkpOworCQly ZXR1cm47CisJfQorCisJLyogVGhpcyBpcyBhIHNtYWxsIGFsbG9jYXRpb24gKG1heGltdW0gfjhL QikgYW5kIFNob3VsZCBOb3QgRmFpbC4gKi8KKwl0YXNrLT50aHJlYWQuc3ZlX3N0YXRlID0KKwkJ a3phbGxvYyhzdmVfc3RhdGVfc2l6ZSh0YXNrKSwgR0ZQX0tFUk5FTCk7CisKKwkvKgorCSAqIElm IGZ1dHVyZSBTVkUgcmV2aXNpb25zIGNhbiBoYXZlIGxhcmdlciB2ZWN0b3JzIHRob3VnaCwKKwkg KiB0aGlzIG1heSBjZWFzZSB0byBiZSB0cnVlOgorCSAqLworCUJVR19PTighdGFzay0+dGhyZWFk LnN2ZV9zdGF0ZSk7Cit9CisKKy8qCisgKiBDYWxsZWQgZnJvbSB0aGUgcHV0X3Rhc2tfc3RydWN0 KCkgcGF0aCwgd2hpY2ggY2Fubm90IGdldCBoZXJlCisgKiB1bmxlc3MgZGVhZF90YXNrIGlzIHJl YWxseSBkZWFkIGFuZCBub3Qgc2NoZWR1bGFibGUuCisgKi8KK3ZvaWQgZnBzaW1kX3JlbGVhc2Vf dGFzayhzdHJ1Y3QgdGFza19zdHJ1Y3QgKmRlYWRfdGFzaykKK3sKKwlfX3N2ZV9mcmVlKGRlYWRf dGFzayk7Cit9CisKKyNlbmRpZiAvKiBDT05GSUdfQVJNNjRfU1ZFICovCisKKy8qCisgKiBUcmFw cGVkIFNWRSBhY2Nlc3MKKyAqCisgKiBTdG9yYWdlIGlzIGFsbG9jYXRlZCBmb3IgdGhlIGZ1bGwg U1ZFIHN0YXRlLCB0aGUgY3VycmVudCBGUFNJTUQKKyAqIHJlZ2lzdGVyIGNvbnRlbnRzIGFyZSBt aWdyYXRlZCBhY3Jvc3MsIGFuZCBUSUZfU1ZFIGlzIHNldCBzbyB0aGF0CisgKiB0aGUgU1ZFIGFj Y2VzcyB0cmFwIHdpbGwgYmUgZGlzYWJsZWQgdGhlIG5leHQgdGltZSB0aGlzIHRhc2sKKyAqIHJl YWNoZXMgcmV0X3RvX3VzZXIuCisgKgorICogVElGX1NWRSBzaG91bGQgYmUgY2xlYXIgb24gZW50 cnk6IG90aGVyd2lzZSwgdGFza19mcHNpbWRfbG9hZCgpCisgKiB3b3VsZCBoYXZlIGRpc2FibGVk IHRoZSBTVkUgYWNjZXNzIHRyYXAgZm9yIHVzZXJzcGFjZSBkdXJpbmcKKyAqIHJldF90b191c2Vy LCBtYWtpbmcgYW4gU1ZFIGFjY2VzcyB0cmFwIGltcG9zc2libGUgaW4gdGhhdCBjYXNlLgorICov Cithc21saW5rYWdlIHZvaWQgZG9fc3ZlX2FjYyh1bnNpZ25lZCBpbnQgZXNyLCBzdHJ1Y3QgcHRf cmVncyAqcmVncykKK3sKKwkvKiBFdmVuIGlmIHdlIGNob3NlIG5vdCB0byB1c2UgU1ZFLCB0aGUg aGFyZHdhcmUgY291bGQgc3RpbGwgdHJhcDogKi8KKwlpZiAodW5saWtlbHkoIXN5c3RlbV9zdXBw b3J0c19zdmUoKSkgfHwgV0FSTl9PTihpc19jb21wYXRfdGFzaygpKSkgeworCQlmb3JjZV9zaWdu YWxfaW5qZWN0KFNJR0lMTCwgSUxMX0lMTE9QQywgcmVncywgMCk7CisJCXJldHVybjsKKwl9CisK KwlzdmVfYWxsb2MoY3VycmVudCk7CisKKwlsb2NhbF9iaF9kaXNhYmxlKCk7CisKKwl0YXNrX2Zw c2ltZF9zYXZlKCk7CisJZnBzaW1kX3RvX3N2ZShjdXJyZW50KTsKKworCS8qIEZvcmNlIHJldF90 b191c2VyIHRvIHJlbG9hZCB0aGUgcmVnaXN0ZXJzOiAqLworCWZwc2ltZF9mbHVzaF90YXNrX3N0 YXRlKGN1cnJlbnQpOworCXNldF90aHJlYWRfZmxhZyhUSUZfRk9SRUlHTl9GUFNUQVRFKTsKKwor CWlmICh0ZXN0X2FuZF9zZXRfdGhyZWFkX2ZsYWcoVElGX1NWRSkpCisJCVdBUk5fT04oMSk7IC8q IFNWRSBhY2Nlc3Mgc2hvdWxkbid0IGhhdmUgdHJhcHBlZCAqLworCisJbG9jYWxfYmhfZW5hYmxl KCk7Cit9CisKKy8qCiAgKiBUcmFwcGVkIEZQL0FTSU1EIGFjY2Vzcy4KICAqLwogYXNtbGlua2Fn ZSB2b2lkIGRvX2Zwc2ltZF9hY2ModW5zaWduZWQgaW50IGVzciwgc3RydWN0IHB0X3JlZ3MgKnJl Z3MpCkBAIC0xNDUsOCArNDI4LDggQEAgdm9pZCBmcHNpbWRfdGhyZWFkX3N3aXRjaChzdHJ1Y3Qg dGFza19zdHJ1Y3QgKm5leHQpCiAJICogdGhlIHJlZ2lzdGVycyBpcyBpbiBmYWN0IHRoZSBtb3N0 IHJlY2VudCB1c2VybGFuZCBGUFNJTUQgc3RhdGUgb2YKIAkgKiAnY3VycmVudCcuCiAJICovCi0J aWYgKGN1cnJlbnQtPm1tICYmICF0ZXN0X3RocmVhZF9mbGFnKFRJRl9GT1JFSUdOX0ZQU1RBVEUp KQotCQlmcHNpbWRfc2F2ZV9zdGF0ZSgmY3VycmVudC0+dGhyZWFkLmZwc2ltZF9zdGF0ZSk7CisJ aWYgKGN1cnJlbnQtPm1tKQorCQl0YXNrX2Zwc2ltZF9zYXZlKCk7CiAKIAlpZiAobmV4dC0+bW0p IHsKIAkJLyoKQEAgLTE2OCw2ICs0NTEsOCBAQCB2b2lkIGZwc2ltZF90aHJlYWRfc3dpdGNoKHN0 cnVjdCB0YXNrX3N0cnVjdCAqbmV4dCkKIAogdm9pZCBmcHNpbWRfZmx1c2hfdGhyZWFkKHZvaWQp CiB7CisJaW50IHZsOworCiAJaWYgKCFzeXN0ZW1fc3VwcG9ydHNfZnBzaW1kKCkpCiAJCXJldHVy bjsKIApAQCAtMTc1LDYgKzQ2MCwzMCBAQCB2b2lkIGZwc2ltZF9mbHVzaF90aHJlYWQodm9pZCkK IAogCW1lbXNldCgmY3VycmVudC0+dGhyZWFkLmZwc2ltZF9zdGF0ZSwgMCwgc2l6ZW9mKHN0cnVj dCBmcHNpbWRfc3RhdGUpKTsKIAlmcHNpbWRfZmx1c2hfdGFza19zdGF0ZShjdXJyZW50KTsKKwor CWlmIChzeXN0ZW1fc3VwcG9ydHNfc3ZlKCkpIHsKKwkJY2xlYXJfdGhyZWFkX2ZsYWcoVElGX1NW RSk7CisJCXN2ZV9mcmVlKGN1cnJlbnQpOworCisJCS8qCisJCSAqIFJlc2V0IHRoZSB0YXNrIHZl Y3RvciBsZW5ndGggYXMgcmVxdWlyZWQuCisJCSAqIFRoaXMgaXMgd2hlcmUgd2UgZW5zdXJlIHRo YXQgYWxsIHVzZXIgdGFza3MgaGF2ZSBhIHZhbGlkCisJCSAqIHZlY3RvciBsZW5ndGggY29uZmln dXJlZDogbm8ga2VybmVsIHRhc2sgY2FuIGJlY29tZSBhIHVzZXIKKwkJICogdGFzayB3aXRob3V0 IGFuIGV4ZWMgYW5kIGhlbmNlIGEgY2FsbCB0byB0aGlzIGZ1bmN0aW9uLgorCQkgKiBJZiBhIGJ1 ZyBjYXVzZXMgdGhpcyB0byBnbyB3cm9uZywgd2UgbWFrZSBzb21lIG5vaXNlIGFuZAorCQkgKiB0 cnkgdG8gZnVkZ2UgdGhyZWFkLnN2ZV92bCB0byBhIHNhZmUgdmFsdWUgaGVyZS4KKwkJICovCisJ CXZsID0gY3VycmVudC0+dGhyZWFkLnN2ZV92bDsKKworCQlpZiAodmwgPT0gMCkKKwkJCXZsID0g U1ZFX1ZMX01JTjsKKworCQlpZiAoV0FSTl9PTighc3ZlX3ZsX3ZhbGlkKHZsKSkpCisJCQl2bCA9 IFNWRV9WTF9NSU47CisKKwkJY3VycmVudC0+dGhyZWFkLnN2ZV92bCA9IHZsOworCX0KKwogCXNl dF90aHJlYWRfZmxhZyhUSUZfRk9SRUlHTl9GUFNUQVRFKTsKIAogCWxvY2FsX2JoX2VuYWJsZSgp OwpAQCAtMTgzLDYgKzQ5Miw5IEBAIHZvaWQgZnBzaW1kX2ZsdXNoX3RocmVhZCh2b2lkKQogLyoK ICAqIFNhdmUgdGhlIHVzZXJsYW5kIEZQU0lNRCBzdGF0ZSBvZiAnY3VycmVudCcgdG8gbWVtb3J5 LCBidXQgb25seSBpZiB0aGUgc3RhdGUKICAqIGN1cnJlbnRseSBoZWxkIGluIHRoZSByZWdpc3Rl cnMgZG9lcyBpbiBmYWN0IGJlbG9uZyB0byAnY3VycmVudCcKKyAqCisgKiBDdXJyZW50bHksIFNW RSB0YXNrcyBjYW4ndCBleGlzdCwgc28ganVzdCBXQVJOIGluIHRoYXQgY2FzZS4KKyAqIFN1YnNl cXVlbnQgcGF0Y2hlcyB3aWxsIGFkZCBmdWxsIFNWRSBzdXBwb3J0IGhlcmUuCiAgKi8KIHZvaWQg ZnBzaW1kX3ByZXNlcnZlX2N1cnJlbnRfc3RhdGUodm9pZCkKIHsKQEAgLTE5NCw2ICs1MDYsOCBA QCB2b2lkIGZwc2ltZF9wcmVzZXJ2ZV9jdXJyZW50X3N0YXRlKHZvaWQpCiAJaWYgKCF0ZXN0X3Ro cmVhZF9mbGFnKFRJRl9GT1JFSUdOX0ZQU1RBVEUpKQogCQlmcHNpbWRfc2F2ZV9zdGF0ZSgmY3Vy cmVudC0+dGhyZWFkLmZwc2ltZF9zdGF0ZSk7CiAKKwlXQVJOX09OX09OQ0UodGVzdF9hbmRfY2xl YXJfdGhyZWFkX2ZsYWcoVElGX1NWRSkpOworCiAJbG9jYWxfYmhfZW5hYmxlKCk7CiB9CiAKQEAg LTIxMiw3ICs1MjYsNyBAQCB2b2lkIGZwc2ltZF9yZXN0b3JlX2N1cnJlbnRfc3RhdGUodm9pZCkK IAlpZiAodGVzdF9hbmRfY2xlYXJfdGhyZWFkX2ZsYWcoVElGX0ZPUkVJR05fRlBTVEFURSkpIHsK IAkJc3RydWN0IGZwc2ltZF9zdGF0ZSAqc3QgPSAmY3VycmVudC0+dGhyZWFkLmZwc2ltZF9zdGF0 ZTsKIAotCQlmcHNpbWRfbG9hZF9zdGF0ZShzdCk7CisJCXRhc2tfZnBzaW1kX2xvYWQoKTsKIAkJ X190aGlzX2NwdV93cml0ZShmcHNpbWRfbGFzdF9zdGF0ZSwgc3QpOwogCQlzdC0+Y3B1ID0gc21w X3Byb2Nlc3Nvcl9pZCgpOwogCX0KQEAgLTM4MSw4ICs2OTUsOCBAQCBzdGF0aWMgaW50IGZwc2lt ZF9jcHVfcG1fbm90aWZpZXIoc3RydWN0IG5vdGlmaWVyX2Jsb2NrICpzZWxmLAogewogCXN3aXRj aCAoY21kKSB7CiAJY2FzZSBDUFVfUE1fRU5URVI6Ci0JCWlmIChjdXJyZW50LT5tbSAmJiAhdGVz dF90aHJlYWRfZmxhZyhUSUZfRk9SRUlHTl9GUFNUQVRFKSkKLQkJCWZwc2ltZF9zYXZlX3N0YXRl KCZjdXJyZW50LT50aHJlYWQuZnBzaW1kX3N0YXRlKTsKKwkJaWYgKGN1cnJlbnQtPm1tKQorCQkJ dGFza19mcHNpbWRfc2F2ZSgpOwogCQl0aGlzX2NwdV93cml0ZShmcHNpbWRfbGFzdF9zdGF0ZSwg TlVMTCk7CiAJCWJyZWFrOwogCWNhc2UgQ1BVX1BNX0VYSVQ6CmRpZmYgLS1naXQgYS9hcmNoL2Fy bTY0L2tlcm5lbC9wcm9jZXNzLmMgYi9hcmNoL2FybTY0L2tlcm5lbC9wcm9jZXNzLmMKaW5kZXgg YzE1ZWM0MS4uYjJhZGNjZSAxMDA2NDQKLS0tIGEvYXJjaC9hcm02NC9rZXJuZWwvcHJvY2Vzcy5j CisrKyBiL2FyY2gvYXJtNjQva2VybmVsL3Byb2Nlc3MuYwpAQCAtNDksNiArNDksNyBAQAogI2lu Y2x1ZGUgPGxpbnV4L25vdGlmaWVyLmg+CiAjaW5jbHVkZSA8dHJhY2UvZXZlbnRzL3Bvd2VyLmg+ CiAjaW5jbHVkZSA8bGludXgvcGVyY3B1Lmg+CisjaW5jbHVkZSA8bGludXgvdGhyZWFkX2luZm8u aD4KIAogI2luY2x1ZGUgPGFzbS9hbHRlcm5hdGl2ZS5oPgogI2luY2x1ZGUgPGFzbS9jb21wYXQu aD4KQEAgLTI3MywxMSArMjc0LDI3IEBAIHZvaWQgcmVsZWFzZV90aHJlYWQoc3RydWN0IHRhc2tf c3RydWN0ICpkZWFkX3Rhc2spCiB7CiB9CiAKK3ZvaWQgYXJjaF9yZWxlYXNlX3Rhc2tfc3RydWN0 KHN0cnVjdCB0YXNrX3N0cnVjdCAqdHNrKQoreworCWZwc2ltZF9yZWxlYXNlX3Rhc2sodHNrKTsK K30KKworLyoKKyAqIHNyYyBhbmQgZHN0IG1heSB0ZW1wb3JhcmlseSBoYXZlIGFsaWFzZWQgc3Zl X3N0YXRlIGFmdGVyIHRhc2tfc3RydWN0CisgKiBpcyBjb3BpZWQuICBXZSBjYW5ub3QgZml4IHRo aXMgcHJvcGVybHkgaGVyZSwgYmVjYXVzZSBzcmMgbWF5IGhhdmUKKyAqIGxpdmUgU1ZFIHN0YXRl IGFuZCBkc3QncyB0aHJlYWRfaW5mbyBtYXkgbm90IGV4aXN0IHlldCwgc28gdHdlYWtpbmcKKyAq IGVpdGhlciBzcmMncyBvciBkc3QncyBUSUZfU1ZFIGlzIG5vdCBzYWZlLgorICoKKyAqIFRoZSB1 bmFsaWFzaW5nIGlzIGRvbmUgaW4gY29weV90aHJlYWQoKSBpbnN0ZWFkLiAgVGhpcyB3b3JrcyBi ZWNhdXNlCisgKiBkc3QgaXMgbm90IHNjaGVkdWxhYmxlIG9yIHRyYWNlYWJsZSB1bnRpbCBib3Ro IG9mIHRoZXNlIGZ1bmN0aW9ucworICogaGF2ZSBiZWVuIGNhbGxlZC4KKyAqLwogaW50IGFyY2hf ZHVwX3Rhc2tfc3RydWN0KHN0cnVjdCB0YXNrX3N0cnVjdCAqZHN0LCBzdHJ1Y3QgdGFza19zdHJ1 Y3QgKnNyYykKIHsKIAlpZiAoY3VycmVudC0+bW0pCiAJCWZwc2ltZF9wcmVzZXJ2ZV9jdXJyZW50 X3N0YXRlKCk7CiAJKmRzdCA9ICpzcmM7CisKIAlyZXR1cm4gMDsKIH0KIApAQCAtMjkwLDYgKzMw NywxMyBAQCBpbnQgY29weV90aHJlYWQodW5zaWduZWQgbG9uZyBjbG9uZV9mbGFncywgdW5zaWdu ZWQgbG9uZyBzdGFja19zdGFydCwKIAogCW1lbXNldCgmcC0+dGhyZWFkLmNwdV9jb250ZXh0LCAw LCBzaXplb2Yoc3RydWN0IGNwdV9jb250ZXh0KSk7CiAKKwkvKgorCSAqIFVuYWxpYXMgcC0+dGhy ZWFkLnN2ZV9zdGF0ZSAoaWYgYW55KSBmcm9tIHRoZSBwYXJlbnQgdGFzaworCSAqIGFuZCBkaXNh YmxlIGRpc2NhcmQgU1ZFIHN0YXRlIGZvciBwOgorCSAqLworCWNsZWFyX3Rza190aHJlYWRfZmxh ZyhwLCBUSUZfU1ZFKTsKKwlwLT50aHJlYWQuc3ZlX3N0YXRlID0gTlVMTDsKKwogCWlmIChsaWtl bHkoIShwLT5mbGFncyAmIFBGX0tUSFJFQUQpKSkgewogCQkqY2hpbGRyZWdzID0gKmN1cnJlbnRf cHRfcmVncygpOwogCQljaGlsZHJlZ3MtPnJlZ3NbMF0gPSAwOwpkaWZmIC0tZ2l0IGEvYXJjaC9h cm02NC9rZXJuZWwvdHJhcHMuYyBiL2FyY2gvYXJtNjQva2VybmVsL3RyYXBzLmMKaW5kZXggMThj MDI5MC4uOWQzYzdmMCAxMDA2NDQKLS0tIGEvYXJjaC9hcm02NC9rZXJuZWwvdHJhcHMuYworKysg Yi9hcmNoL2FybTY0L2tlcm5lbC90cmFwcy5jCkBAIC0zMTAsOCArMzEwLDggQEAgc3RhdGljIGlu dCBjYWxsX3VuZGVmX2hvb2soc3RydWN0IHB0X3JlZ3MgKnJlZ3MpCiAJcmV0dXJuIGZuID8gZm4o cmVncywgaW5zdHIpIDogMTsKIH0KIAotc3RhdGljIHZvaWQgZm9yY2Vfc2lnbmFsX2luamVjdChp bnQgc2lnbmFsLCBpbnQgY29kZSwgc3RydWN0IHB0X3JlZ3MgKnJlZ3MsCi0JCQkJdW5zaWduZWQg bG9uZyBhZGRyZXNzKQordm9pZCBmb3JjZV9zaWduYWxfaW5qZWN0KGludCBzaWduYWwsIGludCBj b2RlLCBzdHJ1Y3QgcHRfcmVncyAqcmVncywKKwkJCSB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCiB7 CiAJc2lnaW5mb190IGluZm87CiAJdm9pZCBfX3VzZXIgKnBjID0gKHZvaWQgX191c2VyICopaW5z dHJ1Y3Rpb25fcG9pbnRlcihyZWdzKTsKQEAgLTMyNSw3ICszMjUsNyBAQCBzdGF0aWMgdm9pZCBm b3JjZV9zaWduYWxfaW5qZWN0KGludCBzaWduYWwsIGludCBjb2RlLCBzdHJ1Y3QgcHRfcmVncyAq cmVncywKIAkJZGVzYyA9ICJpbGxlZ2FsIG1lbW9yeSBhY2Nlc3MiOwogCQlicmVhazsKIAlkZWZh dWx0OgotCQlkZXNjID0gImJhZCBtb2RlIjsKKwkJZGVzYyA9ICJ1bmtub3duIG9yIHVucmVjb3Zl cmFibGUgZXJyb3IiOwogCQlicmVhazsKIAl9CiAKLS0gCjIuMS40CgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwprdm1hcm0gbWFpbGluZyBsaXN0Cmt2bWFy bUBsaXN0cy5jcy5jb2x1bWJpYS5lZHUKaHR0cHM6Ly9saXN0cy5jcy5jb2x1bWJpYS5lZHUvbWFp bG1hbi9saXN0aW5mby9rdm1hcm0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:37668 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753795AbdJaPv5 (ORCPT ); Tue, 31 Oct 2017 11:51:57 -0400 From: Dave Martin Subject: [PATCH v5 13/30] arm64/sve: Core task context handling Date: Tue, 31 Oct 2017 15:51:05 +0000 Message-ID: <1509465082-30427-14-git-send-email-Dave.Martin@arm.com> In-Reply-To: <1509465082-30427-1-git-send-email-Dave.Martin@arm.com> References: <1509465082-30427-1-git-send-email-Dave.Martin@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Szabolcs Nagy , Okamoto Takayuki , kvmarm@lists.cs.columbia.edu, libc-alpha@sourceware.org, linux-arch@vger.kernel.org Message-ID: <20171031155105.DUyK8qVWlb9oa3K-kO3lwMJt8izFgOzQ-MvVYThVyRo@z> This patch adds the core support for switching and managing the SVE architectural state of user tasks. Calls to the existing FPSIMD low-level save/restore functions are factored out as new functions task_fpsimd_{save,load}(), since SVE now dynamically may or may not need to be handled at these points depending on the kernel configuration, hardware features discovered at boot, and the runtime state of the task. To make these decisions as fast as possible, const cpucaps are used where feasible, via the system_supports_sve() helper. The SVE registers are only tracked for threads that have explicitly used SVE, indicated by the new thread flag TIF_SVE. Otherwise, the FPSIMD view of the architectural state is stored in thread.fpsimd_state as usual. When in use, the SVE registers are not stored directly in thread_struct due to their potentially large and variable size. Because the task_struct slab allocator must be configured very early during kernel boot, it is also tricky to configure it correctly to match the maximum vector length provided by the hardware, since this depends on examining secondary CPUs as well as the primary. Instead, a pointer sve_state in thread_struct points to a dynamically allocated buffer containing the SVE register data, and code is added to allocate and free this buffer at appropriate times. TIF_SVE is set when taking an SVE access trap from userspace, if suitable hardware support has been detected. This enables SVE for the thread: a subsequent return to userspace will disable the trap accordingly. If such a trap is taken without sufficient system- wide hardware support, SIGILL is sent to the thread instead as if an undefined instruction had been executed: this may happen if userspace tries to use SVE in a system where not all CPUs support it for example. The kernel will clear TIF_SVE and disable SVE for the thread whenever an explicit syscall is made by userspace. For backwards compatibility reasons and conformance with the spirit of the base AArch64 procedure call standard, the subset of the SVE register state that aliases the FPSIMD registers is still preserved across a syscall even if this happens. The remainder of the SVE register state logically becomes zero at syscall entry, though the actual zeroing work is currently deferred until the thread next tries to use SVE, causing another trap to the kernel. This implementation is suboptimal: in the future, the fastpath case may be optimised to zero the registers in-place and leave SVE enabled for the task, where beneficial. TIF_SVE is also cleared in the following slowpath cases, which are taken as reasonable hints that the task may no longer use SVE: * exec * fork and clone Code is added to sync data between thread.fpsimd_state and thread.sve_state whenever enabling/disabling SVE, in a manner consistent with the SVE architectural programmer's model. Signed-off-by: Dave Martin Reviewed-by: Catalin Marinas Cc: Ard Biesheuvel Cc: Alex BennĂ©e --- Kept Catalin's Reviewed-by, since this is a trivial change. Changes since v4 ---------------- Miscellaneous: * Mark do_sve_acc() as asmlinkage. (Semantic correctness only; no functional impact.) --- arch/arm64/include/asm/fpsimd.h | 16 ++ arch/arm64/include/asm/processor.h | 2 + arch/arm64/include/asm/thread_info.h | 4 + arch/arm64/include/asm/traps.h | 2 + arch/arm64/kernel/entry.S | 39 ++++- arch/arm64/kernel/fpsimd.c | 324 ++++++++++++++++++++++++++++++++++- arch/arm64/kernel/process.c | 24 +++ arch/arm64/kernel/traps.c | 6 +- 8 files changed, 406 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 026a7c7..5655fe1 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -20,6 +20,8 @@ #ifndef __ASSEMBLY__ +#include + /* * FP/SIMD storage area has: * - FPSR and FPCR @@ -72,6 +74,20 @@ extern void sve_load_state(void const *state, u32 const *pfpsr, unsigned long vq_minus_1); extern unsigned int sve_get_vl(void); +#ifdef CONFIG_ARM64_SVE + +extern size_t sve_state_size(struct task_struct const *task); + +extern void sve_alloc(struct task_struct *task); +extern void fpsimd_release_task(struct task_struct *task); + +#else /* ! CONFIG_ARM64_SVE */ + +static inline void sve_alloc(struct task_struct *task) { } +static inline void fpsimd_release_task(struct task_struct *task) { } + +#endif /* ! CONFIG_ARM64_SVE */ + /* For use by EFI runtime services calls only */ extern void __efi_fpsimd_begin(void); extern void __efi_fpsimd_end(void); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 7dddca2..e2f575d 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -105,6 +105,8 @@ struct thread_struct { unsigned long tp2_value; #endif struct fpsimd_state fpsimd_state; + void *sve_state; /* SVE registers, if any */ + unsigned int sve_vl; /* SVE vector length */ unsigned long fault_address; /* fault info */ unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index ddded64..92b7b48 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -63,6 +63,8 @@ struct thread_info { void arch_setup_new_exec(void); #define arch_setup_new_exec arch_setup_new_exec +void arch_release_task_struct(struct task_struct *tsk); + #endif /* @@ -92,6 +94,7 @@ void arch_setup_new_exec(void); #define TIF_RESTORE_SIGMASK 20 #define TIF_SINGLESTEP 21 #define TIF_32BIT 22 /* 32bit process */ +#define TIF_SVE 23 /* Scalable Vector Extension in use */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -105,6 +108,7 @@ void arch_setup_new_exec(void); #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_FSCHECK (1 << TIF_FSCHECK) #define _TIF_32BIT (1 << TIF_32BIT) +#define _TIF_SVE (1 << TIF_SVE) #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h index 45e3da3..1696f9d 100644 --- a/arch/arm64/include/asm/traps.h +++ b/arch/arm64/include/asm/traps.h @@ -34,6 +34,8 @@ struct undef_hook { void register_undef_hook(struct undef_hook *hook); void unregister_undef_hook(struct undef_hook *hook); +void force_signal_inject(int signal, int code, struct pt_regs *regs, + unsigned long address); void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index f5e851e..56e848f 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -607,6 +607,8 @@ el0_sync: b.eq el0_ia cmp x24, #ESR_ELx_EC_FP_ASIMD // FP/ASIMD access b.eq el0_fpsimd_acc + cmp x24, #ESR_ELx_EC_SVE // SVE access + b.eq el0_sve_acc cmp x24, #ESR_ELx_EC_FP_EXC64 // FP/ASIMD exception b.eq el0_fpsimd_exc cmp x24, #ESR_ELx_EC_SYS64 // configurable trap @@ -658,6 +660,7 @@ el0_svc_compat: /* * AArch32 syscall handling */ + ldr x16, [tsk, #TSK_TI_FLAGS] // load thread flags adrp stbl, compat_sys_call_table // load compat syscall table pointer mov wscno, w7 // syscall number in w7 (r7) mov wsc_nr, #__NR_compat_syscalls @@ -705,9 +708,19 @@ el0_fpsimd_acc: mov x1, sp bl do_fpsimd_acc b ret_to_user +el0_sve_acc: + /* + * Scalable Vector Extension access + */ + enable_dbg_and_irq + ct_user_exit + mov x0, x25 + mov x1, sp + bl do_sve_acc + b ret_to_user el0_fpsimd_exc: /* - * Floating Point or Advanced SIMD exception + * Floating Point, Advanced SIMD or SVE exception */ enable_dbg ct_user_exit @@ -835,16 +848,36 @@ ENDPROC(ret_to_user) */ .align 6 el0_svc: + ldr x16, [tsk, #TSK_TI_FLAGS] // load thread flags adrp stbl, sys_call_table // load syscall table pointer mov wscno, w8 // syscall number in w8 mov wsc_nr, #__NR_syscalls + +#ifndef CONFIG_ARM64_SVE + b el0_svc_naked +#else + tbz x16, #TIF_SVE, el0_svc_naked // Skip unless TIF_SVE set: + bic x16, x16, #_TIF_SVE // discard SVE state + str x16, [tsk, #TSK_TI_FLAGS] + + /* + * task_fpsimd_load() won't be called to update CPACR_EL1 in + * ret_to_user unless TIF_FOREIGN_FPSTATE is still set, which only + * happens if a context switch or kernel_neon_begin() or context + * modification (sigreturn, ptrace) intervenes. + * So, ensure that CPACR_EL1 is already correct for the fast-path case: + */ + mrs x9, cpacr_el1 + bic x9, x9, #CPACR_EL1_ZEN_EL0EN // disable SVE for el0 + msr cpacr_el1, x9 // synchronised by eret to el0 +#endif /* CONFIG_ARM64_SVE */ + el0_svc_naked: // compat entry point stp x0, xscno, [sp, #S_ORIG_X0] // save the original x0 and syscall number enable_dbg_and_irq ct_user_exit 1 - ldr x16, [tsk, #TSK_TI_FLAGS] // check for syscall hooks - tst x16, #_TIF_SYSCALL_WORK + tst x16, #_TIF_SYSCALL_WORK // check for syscall hooks b.ne __sys_trace cmp wscno, wsc_nr // check upper syscall limit b.hs ni_sys diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 2baba0d..787f5d3 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -18,19 +18,27 @@ */ #include +#include +#include #include #include #include #include +#include #include #include #include +#include #include #include +#include #include #include #include +#include +#include +#include #define FPEXC_IOF (1 << 0) #define FPEXC_DZF (1 << 1) @@ -40,6 +48,8 @@ #define FPEXC_IDF (1 << 7) /* + * (Note: in this discussion, statements about FPSIMD apply equally to SVE.) + * * In order to reduce the number of times the FPSIMD state is needlessly saved * and restored, we need to keep track of two things: * (a) for each task, we need to remember which CPU was the last one to have @@ -101,6 +111,279 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state); /* + * Call __sve_free() directly only if you know task can't be scheduled + * or preempted. + */ +static void __sve_free(struct task_struct *task) +{ + kfree(task->thread.sve_state); + task->thread.sve_state = NULL; +} + +static void sve_free(struct task_struct *task) +{ + WARN_ON(test_tsk_thread_flag(task, TIF_SVE)); + + __sve_free(task); +} + + +/* Offset of FFR in the SVE register dump */ +static size_t sve_ffr_offset(int vl) +{ + return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET; +} + +static void *sve_pffr(struct task_struct *task) +{ + return (char *)task->thread.sve_state + + sve_ffr_offset(task->thread.sve_vl); +} + +static void change_cpacr(u64 val, u64 mask) +{ + u64 cpacr = read_sysreg(CPACR_EL1); + u64 new = (cpacr & ~mask) | val; + + if (new != cpacr) + write_sysreg(new, CPACR_EL1); +} + +static void sve_user_disable(void) +{ + change_cpacr(0, CPACR_EL1_ZEN_EL0EN); +} + +static void sve_user_enable(void) +{ + change_cpacr(CPACR_EL1_ZEN_EL0EN, CPACR_EL1_ZEN_EL0EN); +} + +/* + * TIF_SVE controls whether a task can use SVE without trapping while + * in userspace, and also the way a task's FPSIMD/SVE state is stored + * in thread_struct. + * + * The kernel uses this flag to track whether a user task is actively + * using SVE, and therefore whether full SVE register state needs to + * be tracked. If not, the cheaper FPSIMD context handling code can + * be used instead of the more costly SVE equivalents. + * + * * TIF_SVE set: + * + * The task can execute SVE instructions while in userspace without + * trapping to the kernel. + * + * When stored, Z0-Z31 (incorporating Vn in bits[127:0] or the + * corresponding Zn), P0-P15 and FFR are encoded in in + * task->thread.sve_state, formatted appropriately for vector + * length task->thread.sve_vl. + * + * task->thread.sve_state must point to a valid buffer at least + * sve_state_size(task) bytes in size. + * + * During any syscall, the kernel may optionally clear TIF_SVE and + * discard the vector state except for the FPSIMD subset. + * + * * TIF_SVE clear: + * + * An attempt by the user task to execute an SVE instruction causes + * do_sve_acc() to be called, which does some preparation and then + * sets TIF_SVE. + * + * When stored, FPSIMD registers V0-V31 are encoded in + * task->fpsimd_state; bits [max : 128] for each of Z0-Z31 are + * logically zero but not stored anywhere; P0-P15 and FFR are not + * stored and have unspecified values from userspace's point of + * view. For hygiene purposes, the kernel zeroes them on next use, + * but userspace is discouraged from relying on this. + * + * task->thread.sve_state does not need to be non-NULL, valid or any + * particular size: it must not be dereferenced. + * + * * FPSR and FPCR are always stored in task->fpsimd_state irrespctive of + * whether TIF_SVE is clear or set, since these are not vector length + * dependent. + */ + +/* + * Update current's FPSIMD/SVE registers from thread_struct. + * + * This function should be called only when the FPSIMD/SVE state in + * thread_struct is known to be up to date, when preparing to enter + * userspace. + * + * Softirqs (and preemption) must be disabled. + */ +static void task_fpsimd_load(void) +{ + WARN_ON(!in_softirq() && !irqs_disabled()); + + if (system_supports_sve() && test_thread_flag(TIF_SVE)) + sve_load_state(sve_pffr(current), + ¤t->thread.fpsimd_state.fpsr, + sve_vq_from_vl(current->thread.sve_vl) - 1); + else + fpsimd_load_state(¤t->thread.fpsimd_state); + + if (system_supports_sve()) { + /* Toggle SVE trapping for userspace if needed */ + if (test_thread_flag(TIF_SVE)) + sve_user_enable(); + else + sve_user_disable(); + + /* Serialised by exception return to user */ + } +} + +/* + * Ensure current's FPSIMD/SVE storage in thread_struct is up to date + * with respect to the CPU registers. + * + * Softirqs (and preemption) must be disabled. + */ +static void task_fpsimd_save(void) +{ + WARN_ON(!in_softirq() && !irqs_disabled()); + + if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { + if (system_supports_sve() && test_thread_flag(TIF_SVE)) { + if (WARN_ON(sve_get_vl() != current->thread.sve_vl)) { + /* + * Can't save the user regs, so current would + * re-enter user with corrupt state. + * There's no way to recover, so kill it: + */ + force_signal_inject( + SIGKILL, 0, current_pt_regs(), 0); + return; + } + + sve_save_state(sve_pffr(current), + ¤t->thread.fpsimd_state.fpsr); + } else + fpsimd_save_state(¤t->thread.fpsimd_state); + } +} + +#define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ + (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) + +/* + * Transfer the FPSIMD state in task->thread.fpsimd_state to + * task->thread.sve_state. + * + * Task can be a non-runnable task, or current. In the latter case, + * softirqs (and preemption) must be disabled. + * task->thread.sve_state must point to at least sve_state_size(task) + * bytes of allocated kernel memory. + * task->thread.fpsimd_state must be up to date before calling this function. + */ +static void fpsimd_to_sve(struct task_struct *task) +{ + unsigned int vq; + void *sst = task->thread.sve_state; + struct fpsimd_state const *fst = &task->thread.fpsimd_state; + unsigned int i; + + if (!system_supports_sve()) + return; + + vq = sve_vq_from_vl(task->thread.sve_vl); + for (i = 0; i < 32; ++i) + memcpy(ZREG(sst, vq, i), &fst->vregs[i], + sizeof(fst->vregs[i])); +} + +#ifdef CONFIG_ARM64_SVE + +/* + * Return how many bytes of memory are required to store the full SVE + * state for task, given task's currently configured vector length. + */ +size_t sve_state_size(struct task_struct const *task) +{ + return SVE_SIG_REGS_SIZE(sve_vq_from_vl(task->thread.sve_vl)); +} + +/* + * Ensure that task->thread.sve_state is allocated and sufficiently large. + * + * This function should be used only in preparation for replacing + * task->thread.sve_state with new data. The memory is always zeroed + * here to prevent stale data from showing through: this is done in + * the interest of testability and predictability: except in the + * do_sve_acc() case, there is no ABI requirement to hide stale data + * written previously be task. + */ +void sve_alloc(struct task_struct *task) +{ + if (task->thread.sve_state) { + memset(task->thread.sve_state, 0, sve_state_size(current)); + return; + } + + /* This is a small allocation (maximum ~8KB) and Should Not Fail. */ + task->thread.sve_state = + kzalloc(sve_state_size(task), GFP_KERNEL); + + /* + * If future SVE revisions can have larger vectors though, + * this may cease to be true: + */ + BUG_ON(!task->thread.sve_state); +} + +/* + * Called from the put_task_struct() path, which cannot get here + * unless dead_task is really dead and not schedulable. + */ +void fpsimd_release_task(struct task_struct *dead_task) +{ + __sve_free(dead_task); +} + +#endif /* CONFIG_ARM64_SVE */ + +/* + * Trapped SVE access + * + * Storage is allocated for the full SVE state, the current FPSIMD + * register contents are migrated across, and TIF_SVE is set so that + * the SVE access trap will be disabled the next time this task + * reaches ret_to_user. + * + * TIF_SVE should be clear on entry: otherwise, task_fpsimd_load() + * would have disabled the SVE access trap for userspace during + * ret_to_user, making an SVE access trap impossible in that case. + */ +asmlinkage void do_sve_acc(unsigned int esr, struct pt_regs *regs) +{ + /* Even if we chose not to use SVE, the hardware could still trap: */ + if (unlikely(!system_supports_sve()) || WARN_ON(is_compat_task())) { + force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0); + return; + } + + sve_alloc(current); + + local_bh_disable(); + + task_fpsimd_save(); + fpsimd_to_sve(current); + + /* Force ret_to_user to reload the registers: */ + fpsimd_flush_task_state(current); + set_thread_flag(TIF_FOREIGN_FPSTATE); + + if (test_and_set_thread_flag(TIF_SVE)) + WARN_ON(1); /* SVE access shouldn't have trapped */ + + local_bh_enable(); +} + +/* * Trapped FP/ASIMD access. */ asmlinkage void do_fpsimd_acc(unsigned int esr, struct pt_regs *regs) @@ -145,8 +428,8 @@ void fpsimd_thread_switch(struct task_struct *next) * the registers is in fact the most recent userland FPSIMD state of * 'current'. */ - if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(¤t->thread.fpsimd_state); + if (current->mm) + task_fpsimd_save(); if (next->mm) { /* @@ -168,6 +451,8 @@ void fpsimd_thread_switch(struct task_struct *next) void fpsimd_flush_thread(void) { + int vl; + if (!system_supports_fpsimd()) return; @@ -175,6 +460,30 @@ void fpsimd_flush_thread(void) memset(¤t->thread.fpsimd_state, 0, sizeof(struct fpsimd_state)); fpsimd_flush_task_state(current); + + if (system_supports_sve()) { + clear_thread_flag(TIF_SVE); + sve_free(current); + + /* + * Reset the task vector length as required. + * This is where we ensure that all user tasks have a valid + * vector length configured: no kernel task can become a user + * task without an exec and hence a call to this function. + * If a bug causes this to go wrong, we make some noise and + * try to fudge thread.sve_vl to a safe value here. + */ + vl = current->thread.sve_vl; + + if (vl == 0) + vl = SVE_VL_MIN; + + if (WARN_ON(!sve_vl_valid(vl))) + vl = SVE_VL_MIN; + + current->thread.sve_vl = vl; + } + set_thread_flag(TIF_FOREIGN_FPSTATE); local_bh_enable(); @@ -183,6 +492,9 @@ void fpsimd_flush_thread(void) /* * Save the userland FPSIMD state of 'current' to memory, but only if the state * currently held in the registers does in fact belong to 'current' + * + * Currently, SVE tasks can't exist, so just WARN in that case. + * Subsequent patches will add full SVE support here. */ void fpsimd_preserve_current_state(void) { @@ -194,6 +506,8 @@ void fpsimd_preserve_current_state(void) if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) fpsimd_save_state(¤t->thread.fpsimd_state); + WARN_ON_ONCE(test_and_clear_thread_flag(TIF_SVE)); + local_bh_enable(); } @@ -212,7 +526,7 @@ void fpsimd_restore_current_state(void) if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) { struct fpsimd_state *st = ¤t->thread.fpsimd_state; - fpsimd_load_state(st); + task_fpsimd_load(); __this_cpu_write(fpsimd_last_state, st); st->cpu = smp_processor_id(); } @@ -381,8 +695,8 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self, { switch (cmd) { case CPU_PM_ENTER: - if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(¤t->thread.fpsimd_state); + if (current->mm) + task_fpsimd_save(); this_cpu_write(fpsimd_last_state, NULL); break; case CPU_PM_EXIT: diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index c15ec41..b2adcce 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include @@ -273,11 +274,27 @@ void release_thread(struct task_struct *dead_task) { } +void arch_release_task_struct(struct task_struct *tsk) +{ + fpsimd_release_task(tsk); +} + +/* + * src and dst may temporarily have aliased sve_state after task_struct + * is copied. We cannot fix this properly here, because src may have + * live SVE state and dst's thread_info may not exist yet, so tweaking + * either src's or dst's TIF_SVE is not safe. + * + * The unaliasing is done in copy_thread() instead. This works because + * dst is not schedulable or traceable until both of these functions + * have been called. + */ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) { if (current->mm) fpsimd_preserve_current_state(); *dst = *src; + return 0; } @@ -290,6 +307,13 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, memset(&p->thread.cpu_context, 0, sizeof(struct cpu_context)); + /* + * Unalias p->thread.sve_state (if any) from the parent task + * and disable discard SVE state for p: + */ + clear_tsk_thread_flag(p, TIF_SVE); + p->thread.sve_state = NULL; + if (likely(!(p->flags & PF_KTHREAD))) { *childregs = *current_pt_regs(); childregs->regs[0] = 0; diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 18c0290..9d3c7f0 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -310,8 +310,8 @@ static int call_undef_hook(struct pt_regs *regs) return fn ? fn(regs, instr) : 1; } -static void force_signal_inject(int signal, int code, struct pt_regs *regs, - unsigned long address) +void force_signal_inject(int signal, int code, struct pt_regs *regs, + unsigned long address) { siginfo_t info; void __user *pc = (void __user *)instruction_pointer(regs); @@ -325,7 +325,7 @@ static void force_signal_inject(int signal, int code, struct pt_regs *regs, desc = "illegal memory access"; break; default: - desc = "bad mode"; + desc = "unknown or unrecoverable error"; break; } -- 2.1.4 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave.Martin@arm.com (Dave Martin) Date: Tue, 31 Oct 2017 15:51:05 +0000 Subject: [PATCH v5 13/30] arm64/sve: Core task context handling In-Reply-To: <1509465082-30427-1-git-send-email-Dave.Martin@arm.com> References: <1509465082-30427-1-git-send-email-Dave.Martin@arm.com> Message-ID: <1509465082-30427-14-git-send-email-Dave.Martin@arm.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org This patch adds the core support for switching and managing the SVE architectural state of user tasks. Calls to the existing FPSIMD low-level save/restore functions are factored out as new functions task_fpsimd_{save,load}(), since SVE now dynamically may or may not need to be handled at these points depending on the kernel configuration, hardware features discovered at boot, and the runtime state of the task. To make these decisions as fast as possible, const cpucaps are used where feasible, via the system_supports_sve() helper. The SVE registers are only tracked for threads that have explicitly used SVE, indicated by the new thread flag TIF_SVE. Otherwise, the FPSIMD view of the architectural state is stored in thread.fpsimd_state as usual. When in use, the SVE registers are not stored directly in thread_struct due to their potentially large and variable size. Because the task_struct slab allocator must be configured very early during kernel boot, it is also tricky to configure it correctly to match the maximum vector length provided by the hardware, since this depends on examining secondary CPUs as well as the primary. Instead, a pointer sve_state in thread_struct points to a dynamically allocated buffer containing the SVE register data, and code is added to allocate and free this buffer at appropriate times. TIF_SVE is set when taking an SVE access trap from userspace, if suitable hardware support has been detected. This enables SVE for the thread: a subsequent return to userspace will disable the trap accordingly. If such a trap is taken without sufficient system- wide hardware support, SIGILL is sent to the thread instead as if an undefined instruction had been executed: this may happen if userspace tries to use SVE in a system where not all CPUs support it for example. The kernel will clear TIF_SVE and disable SVE for the thread whenever an explicit syscall is made by userspace. For backwards compatibility reasons and conformance with the spirit of the base AArch64 procedure call standard, the subset of the SVE register state that aliases the FPSIMD registers is still preserved across a syscall even if this happens. The remainder of the SVE register state logically becomes zero at syscall entry, though the actual zeroing work is currently deferred until the thread next tries to use SVE, causing another trap to the kernel. This implementation is suboptimal: in the future, the fastpath case may be optimised to zero the registers in-place and leave SVE enabled for the task, where beneficial. TIF_SVE is also cleared in the following slowpath cases, which are taken as reasonable hints that the task may no longer use SVE: * exec * fork and clone Code is added to sync data between thread.fpsimd_state and thread.sve_state whenever enabling/disabling SVE, in a manner consistent with the SVE architectural programmer's model. Signed-off-by: Dave Martin Reviewed-by: Catalin Marinas Cc: Ard Biesheuvel Cc: Alex Benn?e --- Kept Catalin's Reviewed-by, since this is a trivial change. Changes since v4 ---------------- Miscellaneous: * Mark do_sve_acc() as asmlinkage. (Semantic correctness only; no functional impact.) --- arch/arm64/include/asm/fpsimd.h | 16 ++ arch/arm64/include/asm/processor.h | 2 + arch/arm64/include/asm/thread_info.h | 4 + arch/arm64/include/asm/traps.h | 2 + arch/arm64/kernel/entry.S | 39 ++++- arch/arm64/kernel/fpsimd.c | 324 ++++++++++++++++++++++++++++++++++- arch/arm64/kernel/process.c | 24 +++ arch/arm64/kernel/traps.c | 6 +- 8 files changed, 406 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 026a7c7..5655fe1 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -20,6 +20,8 @@ #ifndef __ASSEMBLY__ +#include + /* * FP/SIMD storage area has: * - FPSR and FPCR @@ -72,6 +74,20 @@ extern void sve_load_state(void const *state, u32 const *pfpsr, unsigned long vq_minus_1); extern unsigned int sve_get_vl(void); +#ifdef CONFIG_ARM64_SVE + +extern size_t sve_state_size(struct task_struct const *task); + +extern void sve_alloc(struct task_struct *task); +extern void fpsimd_release_task(struct task_struct *task); + +#else /* ! CONFIG_ARM64_SVE */ + +static inline void sve_alloc(struct task_struct *task) { } +static inline void fpsimd_release_task(struct task_struct *task) { } + +#endif /* ! CONFIG_ARM64_SVE */ + /* For use by EFI runtime services calls only */ extern void __efi_fpsimd_begin(void); extern void __efi_fpsimd_end(void); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 7dddca2..e2f575d 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -105,6 +105,8 @@ struct thread_struct { unsigned long tp2_value; #endif struct fpsimd_state fpsimd_state; + void *sve_state; /* SVE registers, if any */ + unsigned int sve_vl; /* SVE vector length */ unsigned long fault_address; /* fault info */ unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index ddded64..92b7b48 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -63,6 +63,8 @@ struct thread_info { void arch_setup_new_exec(void); #define arch_setup_new_exec arch_setup_new_exec +void arch_release_task_struct(struct task_struct *tsk); + #endif /* @@ -92,6 +94,7 @@ void arch_setup_new_exec(void); #define TIF_RESTORE_SIGMASK 20 #define TIF_SINGLESTEP 21 #define TIF_32BIT 22 /* 32bit process */ +#define TIF_SVE 23 /* Scalable Vector Extension in use */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -105,6 +108,7 @@ void arch_setup_new_exec(void); #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_FSCHECK (1 << TIF_FSCHECK) #define _TIF_32BIT (1 << TIF_32BIT) +#define _TIF_SVE (1 << TIF_SVE) #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h index 45e3da3..1696f9d 100644 --- a/arch/arm64/include/asm/traps.h +++ b/arch/arm64/include/asm/traps.h @@ -34,6 +34,8 @@ struct undef_hook { void register_undef_hook(struct undef_hook *hook); void unregister_undef_hook(struct undef_hook *hook); +void force_signal_inject(int signal, int code, struct pt_regs *regs, + unsigned long address); void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index f5e851e..56e848f 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -607,6 +607,8 @@ el0_sync: b.eq el0_ia cmp x24, #ESR_ELx_EC_FP_ASIMD // FP/ASIMD access b.eq el0_fpsimd_acc + cmp x24, #ESR_ELx_EC_SVE // SVE access + b.eq el0_sve_acc cmp x24, #ESR_ELx_EC_FP_EXC64 // FP/ASIMD exception b.eq el0_fpsimd_exc cmp x24, #ESR_ELx_EC_SYS64 // configurable trap @@ -658,6 +660,7 @@ el0_svc_compat: /* * AArch32 syscall handling */ + ldr x16, [tsk, #TSK_TI_FLAGS] // load thread flags adrp stbl, compat_sys_call_table // load compat syscall table pointer mov wscno, w7 // syscall number in w7 (r7) mov wsc_nr, #__NR_compat_syscalls @@ -705,9 +708,19 @@ el0_fpsimd_acc: mov x1, sp bl do_fpsimd_acc b ret_to_user +el0_sve_acc: + /* + * Scalable Vector Extension access + */ + enable_dbg_and_irq + ct_user_exit + mov x0, x25 + mov x1, sp + bl do_sve_acc + b ret_to_user el0_fpsimd_exc: /* - * Floating Point or Advanced SIMD exception + * Floating Point, Advanced SIMD or SVE exception */ enable_dbg ct_user_exit @@ -835,16 +848,36 @@ ENDPROC(ret_to_user) */ .align 6 el0_svc: + ldr x16, [tsk, #TSK_TI_FLAGS] // load thread flags adrp stbl, sys_call_table // load syscall table pointer mov wscno, w8 // syscall number in w8 mov wsc_nr, #__NR_syscalls + +#ifndef CONFIG_ARM64_SVE + b el0_svc_naked +#else + tbz x16, #TIF_SVE, el0_svc_naked // Skip unless TIF_SVE set: + bic x16, x16, #_TIF_SVE // discard SVE state + str x16, [tsk, #TSK_TI_FLAGS] + + /* + * task_fpsimd_load() won't be called to update CPACR_EL1 in + * ret_to_user unless TIF_FOREIGN_FPSTATE is still set, which only + * happens if a context switch or kernel_neon_begin() or context + * modification (sigreturn, ptrace) intervenes. + * So, ensure that CPACR_EL1 is already correct for the fast-path case: + */ + mrs x9, cpacr_el1 + bic x9, x9, #CPACR_EL1_ZEN_EL0EN // disable SVE for el0 + msr cpacr_el1, x9 // synchronised by eret to el0 +#endif /* CONFIG_ARM64_SVE */ + el0_svc_naked: // compat entry point stp x0, xscno, [sp, #S_ORIG_X0] // save the original x0 and syscall number enable_dbg_and_irq ct_user_exit 1 - ldr x16, [tsk, #TSK_TI_FLAGS] // check for syscall hooks - tst x16, #_TIF_SYSCALL_WORK + tst x16, #_TIF_SYSCALL_WORK // check for syscall hooks b.ne __sys_trace cmp wscno, wsc_nr // check upper syscall limit b.hs ni_sys diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 2baba0d..787f5d3 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -18,19 +18,27 @@ */ #include +#include +#include #include #include #include #include +#include #include #include #include +#include #include #include +#include #include #include #include +#include +#include +#include #define FPEXC_IOF (1 << 0) #define FPEXC_DZF (1 << 1) @@ -40,6 +48,8 @@ #define FPEXC_IDF (1 << 7) /* + * (Note: in this discussion, statements about FPSIMD apply equally to SVE.) + * * In order to reduce the number of times the FPSIMD state is needlessly saved * and restored, we need to keep track of two things: * (a) for each task, we need to remember which CPU was the last one to have @@ -101,6 +111,279 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state); /* + * Call __sve_free() directly only if you know task can't be scheduled + * or preempted. + */ +static void __sve_free(struct task_struct *task) +{ + kfree(task->thread.sve_state); + task->thread.sve_state = NULL; +} + +static void sve_free(struct task_struct *task) +{ + WARN_ON(test_tsk_thread_flag(task, TIF_SVE)); + + __sve_free(task); +} + + +/* Offset of FFR in the SVE register dump */ +static size_t sve_ffr_offset(int vl) +{ + return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET; +} + +static void *sve_pffr(struct task_struct *task) +{ + return (char *)task->thread.sve_state + + sve_ffr_offset(task->thread.sve_vl); +} + +static void change_cpacr(u64 val, u64 mask) +{ + u64 cpacr = read_sysreg(CPACR_EL1); + u64 new = (cpacr & ~mask) | val; + + if (new != cpacr) + write_sysreg(new, CPACR_EL1); +} + +static void sve_user_disable(void) +{ + change_cpacr(0, CPACR_EL1_ZEN_EL0EN); +} + +static void sve_user_enable(void) +{ + change_cpacr(CPACR_EL1_ZEN_EL0EN, CPACR_EL1_ZEN_EL0EN); +} + +/* + * TIF_SVE controls whether a task can use SVE without trapping while + * in userspace, and also the way a task's FPSIMD/SVE state is stored + * in thread_struct. + * + * The kernel uses this flag to track whether a user task is actively + * using SVE, and therefore whether full SVE register state needs to + * be tracked. If not, the cheaper FPSIMD context handling code can + * be used instead of the more costly SVE equivalents. + * + * * TIF_SVE set: + * + * The task can execute SVE instructions while in userspace without + * trapping to the kernel. + * + * When stored, Z0-Z31 (incorporating Vn in bits[127:0] or the + * corresponding Zn), P0-P15 and FFR are encoded in in + * task->thread.sve_state, formatted appropriately for vector + * length task->thread.sve_vl. + * + * task->thread.sve_state must point to a valid buffer at least + * sve_state_size(task) bytes in size. + * + * During any syscall, the kernel may optionally clear TIF_SVE and + * discard the vector state except for the FPSIMD subset. + * + * * TIF_SVE clear: + * + * An attempt by the user task to execute an SVE instruction causes + * do_sve_acc() to be called, which does some preparation and then + * sets TIF_SVE. + * + * When stored, FPSIMD registers V0-V31 are encoded in + * task->fpsimd_state; bits [max : 128] for each of Z0-Z31 are + * logically zero but not stored anywhere; P0-P15 and FFR are not + * stored and have unspecified values from userspace's point of + * view. For hygiene purposes, the kernel zeroes them on next use, + * but userspace is discouraged from relying on this. + * + * task->thread.sve_state does not need to be non-NULL, valid or any + * particular size: it must not be dereferenced. + * + * * FPSR and FPCR are always stored in task->fpsimd_state irrespctive of + * whether TIF_SVE is clear or set, since these are not vector length + * dependent. + */ + +/* + * Update current's FPSIMD/SVE registers from thread_struct. + * + * This function should be called only when the FPSIMD/SVE state in + * thread_struct is known to be up to date, when preparing to enter + * userspace. + * + * Softirqs (and preemption) must be disabled. + */ +static void task_fpsimd_load(void) +{ + WARN_ON(!in_softirq() && !irqs_disabled()); + + if (system_supports_sve() && test_thread_flag(TIF_SVE)) + sve_load_state(sve_pffr(current), + ¤t->thread.fpsimd_state.fpsr, + sve_vq_from_vl(current->thread.sve_vl) - 1); + else + fpsimd_load_state(¤t->thread.fpsimd_state); + + if (system_supports_sve()) { + /* Toggle SVE trapping for userspace if needed */ + if (test_thread_flag(TIF_SVE)) + sve_user_enable(); + else + sve_user_disable(); + + /* Serialised by exception return to user */ + } +} + +/* + * Ensure current's FPSIMD/SVE storage in thread_struct is up to date + * with respect to the CPU registers. + * + * Softirqs (and preemption) must be disabled. + */ +static void task_fpsimd_save(void) +{ + WARN_ON(!in_softirq() && !irqs_disabled()); + + if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { + if (system_supports_sve() && test_thread_flag(TIF_SVE)) { + if (WARN_ON(sve_get_vl() != current->thread.sve_vl)) { + /* + * Can't save the user regs, so current would + * re-enter user with corrupt state. + * There's no way to recover, so kill it: + */ + force_signal_inject( + SIGKILL, 0, current_pt_regs(), 0); + return; + } + + sve_save_state(sve_pffr(current), + ¤t->thread.fpsimd_state.fpsr); + } else + fpsimd_save_state(¤t->thread.fpsimd_state); + } +} + +#define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ + (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) + +/* + * Transfer the FPSIMD state in task->thread.fpsimd_state to + * task->thread.sve_state. + * + * Task can be a non-runnable task, or current. In the latter case, + * softirqs (and preemption) must be disabled. + * task->thread.sve_state must point to at least sve_state_size(task) + * bytes of allocated kernel memory. + * task->thread.fpsimd_state must be up to date before calling this function. + */ +static void fpsimd_to_sve(struct task_struct *task) +{ + unsigned int vq; + void *sst = task->thread.sve_state; + struct fpsimd_state const *fst = &task->thread.fpsimd_state; + unsigned int i; + + if (!system_supports_sve()) + return; + + vq = sve_vq_from_vl(task->thread.sve_vl); + for (i = 0; i < 32; ++i) + memcpy(ZREG(sst, vq, i), &fst->vregs[i], + sizeof(fst->vregs[i])); +} + +#ifdef CONFIG_ARM64_SVE + +/* + * Return how many bytes of memory are required to store the full SVE + * state for task, given task's currently configured vector length. + */ +size_t sve_state_size(struct task_struct const *task) +{ + return SVE_SIG_REGS_SIZE(sve_vq_from_vl(task->thread.sve_vl)); +} + +/* + * Ensure that task->thread.sve_state is allocated and sufficiently large. + * + * This function should be used only in preparation for replacing + * task->thread.sve_state with new data. The memory is always zeroed + * here to prevent stale data from showing through: this is done in + * the interest of testability and predictability: except in the + * do_sve_acc() case, there is no ABI requirement to hide stale data + * written previously be task. + */ +void sve_alloc(struct task_struct *task) +{ + if (task->thread.sve_state) { + memset(task->thread.sve_state, 0, sve_state_size(current)); + return; + } + + /* This is a small allocation (maximum ~8KB) and Should Not Fail. */ + task->thread.sve_state = + kzalloc(sve_state_size(task), GFP_KERNEL); + + /* + * If future SVE revisions can have larger vectors though, + * this may cease to be true: + */ + BUG_ON(!task->thread.sve_state); +} + +/* + * Called from the put_task_struct() path, which cannot get here + * unless dead_task is really dead and not schedulable. + */ +void fpsimd_release_task(struct task_struct *dead_task) +{ + __sve_free(dead_task); +} + +#endif /* CONFIG_ARM64_SVE */ + +/* + * Trapped SVE access + * + * Storage is allocated for the full SVE state, the current FPSIMD + * register contents are migrated across, and TIF_SVE is set so that + * the SVE access trap will be disabled the next time this task + * reaches ret_to_user. + * + * TIF_SVE should be clear on entry: otherwise, task_fpsimd_load() + * would have disabled the SVE access trap for userspace during + * ret_to_user, making an SVE access trap impossible in that case. + */ +asmlinkage void do_sve_acc(unsigned int esr, struct pt_regs *regs) +{ + /* Even if we chose not to use SVE, the hardware could still trap: */ + if (unlikely(!system_supports_sve()) || WARN_ON(is_compat_task())) { + force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0); + return; + } + + sve_alloc(current); + + local_bh_disable(); + + task_fpsimd_save(); + fpsimd_to_sve(current); + + /* Force ret_to_user to reload the registers: */ + fpsimd_flush_task_state(current); + set_thread_flag(TIF_FOREIGN_FPSTATE); + + if (test_and_set_thread_flag(TIF_SVE)) + WARN_ON(1); /* SVE access shouldn't have trapped */ + + local_bh_enable(); +} + +/* * Trapped FP/ASIMD access. */ asmlinkage void do_fpsimd_acc(unsigned int esr, struct pt_regs *regs) @@ -145,8 +428,8 @@ void fpsimd_thread_switch(struct task_struct *next) * the registers is in fact the most recent userland FPSIMD state of * 'current'. */ - if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(¤t->thread.fpsimd_state); + if (current->mm) + task_fpsimd_save(); if (next->mm) { /* @@ -168,6 +451,8 @@ void fpsimd_thread_switch(struct task_struct *next) void fpsimd_flush_thread(void) { + int vl; + if (!system_supports_fpsimd()) return; @@ -175,6 +460,30 @@ void fpsimd_flush_thread(void) memset(¤t->thread.fpsimd_state, 0, sizeof(struct fpsimd_state)); fpsimd_flush_task_state(current); + + if (system_supports_sve()) { + clear_thread_flag(TIF_SVE); + sve_free(current); + + /* + * Reset the task vector length as required. + * This is where we ensure that all user tasks have a valid + * vector length configured: no kernel task can become a user + * task without an exec and hence a call to this function. + * If a bug causes this to go wrong, we make some noise and + * try to fudge thread.sve_vl to a safe value here. + */ + vl = current->thread.sve_vl; + + if (vl == 0) + vl = SVE_VL_MIN; + + if (WARN_ON(!sve_vl_valid(vl))) + vl = SVE_VL_MIN; + + current->thread.sve_vl = vl; + } + set_thread_flag(TIF_FOREIGN_FPSTATE); local_bh_enable(); @@ -183,6 +492,9 @@ void fpsimd_flush_thread(void) /* * Save the userland FPSIMD state of 'current' to memory, but only if the state * currently held in the registers does in fact belong to 'current' + * + * Currently, SVE tasks can't exist, so just WARN in that case. + * Subsequent patches will add full SVE support here. */ void fpsimd_preserve_current_state(void) { @@ -194,6 +506,8 @@ void fpsimd_preserve_current_state(void) if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) fpsimd_save_state(¤t->thread.fpsimd_state); + WARN_ON_ONCE(test_and_clear_thread_flag(TIF_SVE)); + local_bh_enable(); } @@ -212,7 +526,7 @@ void fpsimd_restore_current_state(void) if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) { struct fpsimd_state *st = ¤t->thread.fpsimd_state; - fpsimd_load_state(st); + task_fpsimd_load(); __this_cpu_write(fpsimd_last_state, st); st->cpu = smp_processor_id(); } @@ -381,8 +695,8 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self, { switch (cmd) { case CPU_PM_ENTER: - if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(¤t->thread.fpsimd_state); + if (current->mm) + task_fpsimd_save(); this_cpu_write(fpsimd_last_state, NULL); break; case CPU_PM_EXIT: diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index c15ec41..b2adcce 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include @@ -273,11 +274,27 @@ void release_thread(struct task_struct *dead_task) { } +void arch_release_task_struct(struct task_struct *tsk) +{ + fpsimd_release_task(tsk); +} + +/* + * src and dst may temporarily have aliased sve_state after task_struct + * is copied. We cannot fix this properly here, because src may have + * live SVE state and dst's thread_info may not exist yet, so tweaking + * either src's or dst's TIF_SVE is not safe. + * + * The unaliasing is done in copy_thread() instead. This works because + * dst is not schedulable or traceable until both of these functions + * have been called. + */ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) { if (current->mm) fpsimd_preserve_current_state(); *dst = *src; + return 0; } @@ -290,6 +307,13 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, memset(&p->thread.cpu_context, 0, sizeof(struct cpu_context)); + /* + * Unalias p->thread.sve_state (if any) from the parent task + * and disable discard SVE state for p: + */ + clear_tsk_thread_flag(p, TIF_SVE); + p->thread.sve_state = NULL; + if (likely(!(p->flags & PF_KTHREAD))) { *childregs = *current_pt_regs(); childregs->regs[0] = 0; diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 18c0290..9d3c7f0 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -310,8 +310,8 @@ static int call_undef_hook(struct pt_regs *regs) return fn ? fn(regs, instr) : 1; } -static void force_signal_inject(int signal, int code, struct pt_regs *regs, - unsigned long address) +void force_signal_inject(int signal, int code, struct pt_regs *regs, + unsigned long address) { siginfo_t info; void __user *pc = (void __user *)instruction_pointer(regs); @@ -325,7 +325,7 @@ static void force_signal_inject(int signal, int code, struct pt_regs *regs, desc = "illegal memory access"; break; default: - desc = "bad mode"; + desc = "unknown or unrecoverable error"; break; } -- 2.1.4