From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com (ale.deltatee.com [207.54.116.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 0BE7B21962301 for ; Wed, 12 Sep 2018 17:12:02 -0700 (PDT) From: Logan Gunthorpe Date: Wed, 12 Sep 2018 18:11:44 -0600 Message-Id: <20180913001156.4115-2-logang@deltatee.com> In-Reply-To: <20180913001156.4115-1-logang@deltatee.com> References: <20180913001156.4115-1-logang@deltatee.com> MIME-Version: 1.0 Subject: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Jens Axboe , =?UTF-8?q?Christian=20K=C3=B6nig?= , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: U29tZSBQQ0kgZGV2aWNlcyBtYXkgaGF2ZSBtZW1vcnkgbWFwcGVkIGluIGEgQkFSIHNwYWNlIHRo YXQncwppbnRlbmRlZCBmb3IgdXNlIGluIHBlZXItdG8tcGVlciB0cmFuc2FjdGlvbnMuIEluIG9y ZGVyIHRvIGVuYWJsZQpzdWNoIHRyYW5zYWN0aW9ucyB0aGUgbWVtb3J5IG11c3QgYmUgcmVnaXN0 ZXJlZCB3aXRoIFpPTkVfREVWSUNFIHBhZ2VzCnNvIGl0IGNhbiBiZSB1c2VkIGJ5IERNQSBpbnRl cmZhY2VzIGluIGV4aXN0aW5nIGRyaXZlcnMuCgpBZGQgYW4gaW50ZXJmYWNlIGZvciBvdGhlciBz dWJzeXN0ZW1zIHRvIGZpbmQgYW5kIGFsbG9jYXRlIGNodW5rcyBvZiBQMlAKbWVtb3J5IGFzIG5l Y2Vzc2FyeSB0byBmYWNpbGl0YXRlIHRyYW5zZmVycyBiZXR3ZWVuIHR3byBQQ0kgcGVlcnM6Cgpp bnQgcGNpX3AycGRtYV9hZGRfY2xpZW50KCk7CnN0cnVjdCBwY2lfZGV2ICpwY2lfcDJwbWVtX2Zp bmQoKTsKdm9pZCAqcGNpX2FsbG9jX3AycG1lbSgpOwoKVGhlIG5ldyBpbnRlcmZhY2UgcmVxdWly ZXMgYSBkcml2ZXIgdG8gY29sbGVjdCBhIGxpc3Qgb2YgY2xpZW50IGRldmljZXMKaW52b2x2ZWQg aW4gdGhlIHRyYW5zYWN0aW9uIHdpdGggdGhlIHBjaV9wMnBtZW1fYWRkX2NsaWVudCooKSBmdW5j dGlvbnMKdGhlbiBjYWxsIHBjaV9wMnBtZW1fZmluZCgpIHRvIG9idGFpbiBhbnkgc3VpdGFibGUg UDJQIG1lbW9yeS4gT25jZQp0aGlzIGlzIGRvbmUgdGhlIGxpc3QgaXMgYm91bmQgdG8gdGhlIG1l bW9yeSBhbmQgdGhlIGNhbGxpbmcgZHJpdmVyIGlzCmZyZWUgdG8gYWRkIGFuZCByZW1vdmUgY2xp ZW50cyBhcyBuZWNlc3NhcnkgKGFkZGluZyBpbmNvbXBhdGlibGUgY2xpZW50cwp3aWxsIGZhaWwp LiBXaXRoIGEgc3VpdGFibGUgcDJwbWVtIGRldmljZSwgbWVtb3J5IGNhbiB0aGVuIGJlCmFsbG9j YXRlZCB3aXRoIHBjaV9hbGxvY19wMnBtZW0oKSBmb3IgdXNlIGluIERNQSB0cmFuc2FjdGlvbnMu CgpEZXBlbmRpbmcgb24gaGFyZHdhcmUsIHVzaW5nIHBlZXItdG8tcGVlciBtZW1vcnkgbWF5IHJl ZHVjZSB0aGUgYmFuZHdpZHRoCm9mIHRoZSB0cmFuc2ZlciBidXQgY2FuIHNpZ25pZmljYW50bHkg cmVkdWNlIHByZXNzdXJlIG9uIHN5c3RlbSBtZW1vcnkuClRoaXMgbWF5IGJlIGRlc2lyYWJsZSBp biBtYW55IGNhc2VzOiBmb3IgZXhhbXBsZSBhIHN5c3RlbSBjb3VsZCBiZSBkZXNpZ25lZAp3aXRo IGEgc21hbGwgQ1BVIGNvbm5lY3RlZCB0byBhIFBDSWUgc3dpdGNoIGJ5IGEgc21hbGwgbnVtYmVy IG9mIGxhbmVzCndoaWNoIHdvdWxkIG1heGltaXplIHRoZSBudW1iZXIgb2YgbGFuZXMgYXZhaWxh YmxlIHRvIGNvbm5lY3QgdG8gTlZNZQpkZXZpY2VzLgoKVGhlIGNvZGUgaXMgZGVzaWduZWQgdG8g b25seSB1dGlsaXplIHRoZSBwMnBtZW0gZGV2aWNlIGlmIGFsbCB0aGUgZGV2aWNlcwppbnZvbHZl ZCBpbiBhIHRyYW5zZmVyIGFyZSBiZWhpbmQgdGhlIHNhbWUgUENJIGJyaWRnZS4gVGhpcyBpcyBi ZWNhdXNlIHdlCmhhdmUgbm8gd2F5IG9mIGtub3dpbmcgd2hldGhlciBwZWVyLXRvLXBlZXIgcm91 dGluZyBiZXR3ZWVuIFBDSWUgUm9vdCBQb3J0cwppcyBzdXBwb3J0ZWQgKFBDSWUgcjQuMCwgc2Vj IDEuMy4xKS4gQWRkaXRpb25hbGx5LCB0aGUgYmVuZWZpdHMgb2YgUDJQCnRyYW5zZmVycyB0aGF0 IGdvIHRocm91Z2ggdGhlIFJDIGlzIGxpbWl0ZWQgdG8gb25seSByZWR1Y2luZyBEUkFNIHVzYWdl CmFuZCwgaW4gc29tZSBjYXNlcywgY29kaW5nIGNvbnZlbmllbmNlLiBUaGUgUENJLVNJRyBtYXkg YmUgZXhwbG9yaW5nCmFkZGluZyBhIG5ldyBjYXBhYmlsaXR5IGJpdCB0byBhZHZlcnRpc2Ugd2hl dGhlciB0aGlzIGlzIHBvc3NpYmxlIGZvcgpmdXR1cmUgaGFyZHdhcmUuCgpUaGlzIGNvbW1pdCBp bmNsdWRlcyBzaWduaWZpY2FudCByZXdvcmsgYW5kIGZlZWRiYWNrIGZyb20gQ2hyaXN0b3BoCkhl bGx3aWcuCgpTaWduZWQtb2ZmLWJ5OiBDaHJpc3RvcGggSGVsbHdpZyA8aGNoQGxzdC5kZT4KU2ln bmVkLW9mZi1ieTogTG9nYW4gR3VudGhvcnBlIDxsb2dhbmdAZGVsdGF0ZWUuY29tPgotLS0KIGRy aXZlcnMvcGNpL0tjb25maWcgICAgICAgIHwgIDE3ICsKIGRyaXZlcnMvcGNpL01ha2VmaWxlICAg ICAgIHwgICAxICsKIGRyaXZlcnMvcGNpL3AycGRtYS5jICAgICAgIHwgNzYxICsrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysKIGluY2x1ZGUvbGludXgvbWVtcmVtYXAuaCAgIHwg ICA1ICsKIGluY2x1ZGUvbGludXgvbW0uaCAgICAgICAgIHwgIDE4ICsKIGluY2x1ZGUvbGludXgv cGNpLXAycGRtYS5oIHwgMTAyICsrKysrCiBpbmNsdWRlL2xpbnV4L3BjaS5oICAgICAgICB8ICAg NCArCiA3IGZpbGVzIGNoYW5nZWQsIDkwOCBpbnNlcnRpb25zKCspCiBjcmVhdGUgbW9kZSAxMDA2 NDQgZHJpdmVycy9wY2kvcDJwZG1hLmMKIGNyZWF0ZSBtb2RlIDEwMDY0NCBpbmNsdWRlL2xpbnV4 L3BjaS1wMnBkbWEuaAoKZGlmZiAtLWdpdCBhL2RyaXZlcnMvcGNpL0tjb25maWcgYi9kcml2ZXJz L3BjaS9LY29uZmlnCmluZGV4IDU2ZmY4ZjZkMzFmYy4uZGViNjhiZTRmZGFjIDEwMDY0NAotLS0g YS9kcml2ZXJzL3BjaS9LY29uZmlnCisrKyBiL2RyaXZlcnMvcGNpL0tjb25maWcKQEAgLTEzMiw2 ICsxMzIsMjMgQEAgY29uZmlnIFBDSV9QQVNJRAogCiAJICBJZiB1bnN1cmUsIHNheSBOLgogCitj b25maWcgUENJX1AyUERNQQorCWJvb2wgIlBDSSBwZWVyLXRvLXBlZXIgdHJhbnNmZXIgc3VwcG9y dCIKKwlkZXBlbmRzIG9uIFBDSSAmJiBaT05FX0RFVklDRQorCXNlbGVjdCBHRU5FUklDX0FMTE9D QVRPUgorCWhlbHAKKwkgIEVuYWJsZdGVIGRyaXZlcnMgdG8gZG8gUENJIHBlZXItdG8tcGVlciB0 cmFuc2FjdGlvbnMgdG8gYW5kIGZyb20KKwkgIEJBUnMgdGhhdCBhcmUgZXhwb3NlZCBpbiBvdGhl ciBkZXZpY2VzIHRoYXQgYXJlIHRoZSBwYXJ0IG9mCisJICB0aGUgaGllcmFyY2h5IHdoZXJlIHBl ZXItdG8tcGVlciBETUEgaXMgZ3VhcmFudGVlZCBieSB0aGUgUENJCisJICBzcGVjaWZpY2F0aW9u IHRvIHdvcmsgKGllLiBhbnl0aGluZyBiZWxvdyBhIHNpbmdsZSBQQ0kgYnJpZGdlKS4KKworCSAg TWFueSBQQ0llIHJvb3QgY29tcGxleGVzIGRvIG5vdCBzdXBwb3J0IFAyUCB0cmFuc2FjdGlvbnMg YW5kCisJICBpdCdzIGhhcmQgdG8gdGVsbCB3aGljaCBzdXBwb3J0IGl0IGF0IGFsbCwgc28gYXQg dGhpcyB0aW1lLAorCSAgUDJQIERNQSB0cmFuc2F0aW9ucyBtdXN0IGJlIGJldHdlZW4gZGV2aWNl cyBiZWhpbmQgdGhlIHNhbWUgcm9vdAorCSAgcG9ydC4KKworCSAgSWYgdW5zdXJlLCBzYXkgTi4K KwogY29uZmlnIFBDSV9MQUJFTAogCWRlZl9ib29sIHkgaWYgKERNSSB8fCBBQ1BJKQogCWRlcGVu ZHMgb24gUENJCmRpZmYgLS1naXQgYS9kcml2ZXJzL3BjaS9NYWtlZmlsZSBiL2RyaXZlcnMvcGNp L01ha2VmaWxlCmluZGV4IDFiMmNmZTUxZThkNy4uODVmNGE3MDNiMmJlIDEwMDY0NAotLS0gYS9k cml2ZXJzL3BjaS9NYWtlZmlsZQorKysgYi9kcml2ZXJzL3BjaS9NYWtlZmlsZQpAQCAtMjYsNiAr MjYsNyBAQCBvYmotJChDT05GSUdfUENJX1NZU0NBTEwpCSs9IHN5c2NhbGwubwogb2JqLSQoQ09O RklHX1BDSV9TVFVCKQkJKz0gcGNpLXN0dWIubwogb2JqLSQoQ09ORklHX1BDSV9QRl9TVFVCKQkr PSBwY2ktcGYtc3R1Yi5vCiBvYmotJChDT05GSUdfUENJX0VDQU0pCQkrPSBlY2FtLm8KK29iai0k KENPTkZJR19QQ0lfUDJQRE1BKQkrPSBwMnBkbWEubwogb2JqLSQoQ09ORklHX1hFTl9QQ0lERVZf RlJPTlRFTkQpICs9IHhlbi1wY2lmcm9udC5vCiAKICMgRW5kcG9pbnQgbGlicmFyeSBtdXN0IGJl IGluaXRpYWxpemVkIGJlZm9yZSBpdHMgdXNlcnMKZGlmZiAtLWdpdCBhL2RyaXZlcnMvcGNpL3Ay cGRtYS5jIGIvZHJpdmVycy9wY2kvcDJwZG1hLmMKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXgg MDAwMDAwMDAwMDAwLi44OGFhZWM1MzUxY2QKLS0tIC9kZXYvbnVsbAorKysgYi9kcml2ZXJzL3Bj aS9wMnBkbWEuYwpAQCAtMCwwICsxLDc2MSBAQAorLy8gU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6 IEdQTC0yLjAKKy8qCisgKiBQQ0kgUGVlciAyIFBlZXIgRE1BIHN1cHBvcnQuCisgKgorICogQ29w eXJpZ2h0IChjKSAyMDE2LTIwMTgsIExvZ2FuIEd1bnRob3JwZQorICogQ29weXJpZ2h0IChjKSAy MDE2LTIwMTcsIE1pY3Jvc2VtaSBDb3Jwb3JhdGlvbgorICogQ29weXJpZ2h0IChjKSAyMDE3LCBD aHJpc3RvcGggSGVsbHdpZworICogQ29weXJpZ2h0IChjKSAyMDE4LCBFaWRldGljb20gSW5jLgor ICovCisKKyNkZWZpbmUgcHJfZm10KGZtdCkgInBjaS1wMnBkbWE6ICIgZm10CisjaW5jbHVkZSA8 bGludXgvcGNpLXAycGRtYS5oPgorI2luY2x1ZGUgPGxpbnV4L21vZHVsZS5oPgorI2luY2x1ZGUg PGxpbnV4L3NsYWIuaD4KKyNpbmNsdWRlIDxsaW51eC9nZW5hbGxvYy5oPgorI2luY2x1ZGUgPGxp bnV4L21lbXJlbWFwLmg+CisjaW5jbHVkZSA8bGludXgvcGVyY3B1LXJlZmNvdW50Lmg+CisjaW5j bHVkZSA8bGludXgvcmFuZG9tLmg+CisjaW5jbHVkZSA8bGludXgvc2VxX2J1Zi5oPgorCitzdHJ1 Y3QgcGNpX3AycGRtYSB7CisJc3RydWN0IHBlcmNwdV9yZWYgZGV2bWFwX3JlZjsKKwlzdHJ1Y3Qg Y29tcGxldGlvbiBkZXZtYXBfcmVmX2RvbmU7CisJc3RydWN0IGdlbl9wb29sICpwb29sOworCWJv b2wgcDJwbWVtX3B1Ymxpc2hlZDsKK307CisKK3N0YXRpYyB2b2lkIHBjaV9wMnBkbWFfcGVyY3B1 X3JlbGVhc2Uoc3RydWN0IHBlcmNwdV9yZWYgKnJlZikKK3sKKwlzdHJ1Y3QgcGNpX3AycGRtYSAq cDJwID0KKwkJY29udGFpbmVyX29mKHJlZiwgc3RydWN0IHBjaV9wMnBkbWEsIGRldm1hcF9yZWYp OworCisJY29tcGxldGVfYWxsKCZwMnAtPmRldm1hcF9yZWZfZG9uZSk7Cit9CisKK3N0YXRpYyB2 b2lkIHBjaV9wMnBkbWFfcGVyY3B1X2tpbGwodm9pZCAqZGF0YSkKK3sKKwlzdHJ1Y3QgcGVyY3B1 X3JlZiAqcmVmID0gZGF0YTsKKworCWlmIChwZXJjcHVfcmVmX2lzX2R5aW5nKHJlZikpCisJCXJl dHVybjsKKworCXBlcmNwdV9yZWZfa2lsbChyZWYpOworfQorCitzdGF0aWMgdm9pZCBwY2lfcDJw ZG1hX3JlbGVhc2Uodm9pZCAqZGF0YSkKK3sKKwlzdHJ1Y3QgcGNpX2RldiAqcGRldiA9IGRhdGE7 CisKKwlpZiAoIXBkZXYtPnAycGRtYSkKKwkJcmV0dXJuOworCisJd2FpdF9mb3JfY29tcGxldGlv bigmcGRldi0+cDJwZG1hLT5kZXZtYXBfcmVmX2RvbmUpOworCXBlcmNwdV9yZWZfZXhpdCgmcGRl di0+cDJwZG1hLT5kZXZtYXBfcmVmKTsKKworCWdlbl9wb29sX2Rlc3Ryb3kocGRldi0+cDJwZG1h LT5wb29sKTsKKwlwZGV2LT5wMnBkbWEgPSBOVUxMOworfQorCitzdGF0aWMgaW50IHBjaV9wMnBk bWFfc2V0dXAoc3RydWN0IHBjaV9kZXYgKnBkZXYpCit7CisJaW50IGVycm9yID0gLUVOT01FTTsK KwlzdHJ1Y3QgcGNpX3AycGRtYSAqcDJwOworCisJcDJwID0gZGV2bV9remFsbG9jKCZwZGV2LT5k ZXYsIHNpemVvZigqcDJwKSwgR0ZQX0tFUk5FTCk7CisJaWYgKCFwMnApCisJCXJldHVybiAtRU5P TUVNOworCisJcDJwLT5wb29sID0gZ2VuX3Bvb2xfY3JlYXRlKFBBR0VfU0hJRlQsIGRldl90b19u b2RlKCZwZGV2LT5kZXYpKTsKKwlpZiAoIXAycC0+cG9vbCkKKwkJZ290byBvdXQ7CisKKwlpbml0 X2NvbXBsZXRpb24oJnAycC0+ZGV2bWFwX3JlZl9kb25lKTsKKwllcnJvciA9IHBlcmNwdV9yZWZf aW5pdCgmcDJwLT5kZXZtYXBfcmVmLAorCQkJcGNpX3AycGRtYV9wZXJjcHVfcmVsZWFzZSwgMCwg R0ZQX0tFUk5FTCk7CisJaWYgKGVycm9yKQorCQlnb3RvIG91dF9wb29sX2Rlc3Ryb3k7CisKKwlw ZXJjcHVfcmVmX3N3aXRjaF90b19hdG9taWNfc3luYygmcDJwLT5kZXZtYXBfcmVmKTsKKworCWVy cm9yID0gZGV2bV9hZGRfYWN0aW9uX29yX3Jlc2V0KCZwZGV2LT5kZXYsIHBjaV9wMnBkbWFfcmVs ZWFzZSwgcGRldik7CisJaWYgKGVycm9yKQorCQlnb3RvIG91dF9wb29sX2Rlc3Ryb3k7CisKKwlw ZGV2LT5wMnBkbWEgPSBwMnA7CisKKwlyZXR1cm4gMDsKKworb3V0X3Bvb2xfZGVzdHJveToKKwln ZW5fcG9vbF9kZXN0cm95KHAycC0+cG9vbCk7CitvdXQ6CisJZGV2bV9rZnJlZSgmcGRldi0+ZGV2 LCBwMnApOworCXJldHVybiBlcnJvcjsKK30KKworLyoqCisgKiBwY2lfcDJwZG1hX2FkZF9yZXNv dXJjZSAtIGFkZCBtZW1vcnkgZm9yIHVzZSBhcyBwMnAgbWVtb3J5CisgKiBAcGRldjogdGhlIGRl dmljZSB0byBhZGQgdGhlIG1lbW9yeSB0bworICogQGJhcjogUENJIEJBUiB0byBhZGQKKyAqIEBz aXplOiBzaXplIG9mIHRoZSBtZW1vcnkgdG8gYWRkLCBtYXkgYmUgemVybyB0byB1c2UgdGhlIHdo b2xlIEJBUgorICogQG9mZnNldDogb2Zmc2V0IGludG8gdGhlIFBDSSBCQVIKKyAqCisgKiBUaGUg bWVtb3J5IHdpbGwgYmUgZ2l2ZW4gWk9ORV9ERVZJQ0Ugc3RydWN0IHBhZ2VzIHNvIHRoYXQgaXQg bWF5CisgKiBiZSB1c2VkIHdpdGggYW55IERNQSByZXF1ZXN0LgorICovCitpbnQgcGNpX3AycGRt YV9hZGRfcmVzb3VyY2Uoc3RydWN0IHBjaV9kZXYgKnBkZXYsIGludCBiYXIsIHNpemVfdCBzaXpl LAorCQkJICAgIHU2NCBvZmZzZXQpCit7CisJc3RydWN0IGRldl9wYWdlbWFwICpwZ21hcDsKKwl2 b2lkICphZGRyOworCWludCBlcnJvcjsKKworCWlmICghKHBjaV9yZXNvdXJjZV9mbGFncyhwZGV2 LCBiYXIpICYgSU9SRVNPVVJDRV9NRU0pKQorCQlyZXR1cm4gLUVJTlZBTDsKKworCWlmIChvZmZz ZXQgPj0gcGNpX3Jlc291cmNlX2xlbihwZGV2LCBiYXIpKQorCQlyZXR1cm4gLUVJTlZBTDsKKwor CWlmICghc2l6ZSkKKwkJc2l6ZSA9IHBjaV9yZXNvdXJjZV9sZW4ocGRldiwgYmFyKSAtIG9mZnNl dDsKKworCWlmIChzaXplICsgb2Zmc2V0ID4gcGNpX3Jlc291cmNlX2xlbihwZGV2LCBiYXIpKQor CQlyZXR1cm4gLUVJTlZBTDsKKworCWlmICghcGRldi0+cDJwZG1hKSB7CisJCWVycm9yID0gcGNp X3AycGRtYV9zZXR1cChwZGV2KTsKKwkJaWYgKGVycm9yKQorCQkJcmV0dXJuIGVycm9yOworCX0K KworCXBnbWFwID0gZGV2bV9remFsbG9jKCZwZGV2LT5kZXYsIHNpemVvZigqcGdtYXApLCBHRlBf S0VSTkVMKTsKKwlpZiAoIXBnbWFwKQorCQlyZXR1cm4gLUVOT01FTTsKKworCXBnbWFwLT5yZXMu c3RhcnQgPSBwY2lfcmVzb3VyY2Vfc3RhcnQocGRldiwgYmFyKSArIG9mZnNldDsKKwlwZ21hcC0+ cmVzLmVuZCA9IHBnbWFwLT5yZXMuc3RhcnQgKyBzaXplIC0gMTsKKwlwZ21hcC0+cmVzLmZsYWdz ID0gcGNpX3Jlc291cmNlX2ZsYWdzKHBkZXYsIGJhcik7CisJcGdtYXAtPnJlZiA9ICZwZGV2LT5w MnBkbWEtPmRldm1hcF9yZWY7CisJcGdtYXAtPnR5cGUgPSBNRU1PUllfREVWSUNFX1BDSV9QMlBE TUE7CisKKwlhZGRyID0gZGV2bV9tZW1yZW1hcF9wYWdlcygmcGRldi0+ZGV2LCBwZ21hcCk7CisJ aWYgKElTX0VSUihhZGRyKSkgeworCQllcnJvciA9IFBUUl9FUlIoYWRkcik7CisJCWdvdG8gcGdt YXBfZnJlZTsKKwl9CisKKwllcnJvciA9IGdlbl9wb29sX2FkZF92aXJ0KHBkZXYtPnAycGRtYS0+ cG9vbCwgKHVuc2lnbmVkIGxvbmcpYWRkciwKKwkJCXBjaV9idXNfYWRkcmVzcyhwZGV2LCBiYXIp ICsgb2Zmc2V0LAorCQkJcmVzb3VyY2Vfc2l6ZSgmcGdtYXAtPnJlcyksIGRldl90b19ub2RlKCZw ZGV2LT5kZXYpKTsKKwlpZiAoZXJyb3IpCisJCWdvdG8gcGdtYXBfZnJlZTsKKworCWVycm9yID0g ZGV2bV9hZGRfYWN0aW9uX29yX3Jlc2V0KCZwZGV2LT5kZXYsIHBjaV9wMnBkbWFfcGVyY3B1X2tp bGwsCisJCQkJCSAgJnBkZXYtPnAycGRtYS0+ZGV2bWFwX3JlZik7CisJaWYgKGVycm9yKQorCQln b3RvIHBnbWFwX2ZyZWU7CisKKwlwY2lfaW5mbyhwZGV2LCAiYWRkZWQgcGVlci10by1wZWVyIERN QSBtZW1vcnkgJXBSXG4iLAorCQkgJnBnbWFwLT5yZXMpOworCisJcmV0dXJuIDA7CisKK3BnbWFw X2ZyZWU6CisJZGV2cmVzX2ZyZWUocGdtYXApOworCXJldHVybiBlcnJvcjsKK30KK0VYUE9SVF9T WU1CT0xfR1BMKHBjaV9wMnBkbWFfYWRkX3Jlc291cmNlKTsKKworc3RhdGljIHN0cnVjdCBwY2lf ZGV2ICpmaW5kX3BhcmVudF9wY2lfZGV2KHN0cnVjdCBkZXZpY2UgKmRldikKK3sKKwlzdHJ1Y3Qg ZGV2aWNlICpwYXJlbnQ7CisKKwlkZXYgPSBnZXRfZGV2aWNlKGRldik7CisKKwl3aGlsZSAoZGV2 KSB7CisJCWlmIChkZXZfaXNfcGNpKGRldikpCisJCQlyZXR1cm4gdG9fcGNpX2RldihkZXYpOwor CisJCXBhcmVudCA9IGdldF9kZXZpY2UoZGV2LT5wYXJlbnQpOworCQlwdXRfZGV2aWNlKGRldik7 CisJCWRldiA9IHBhcmVudDsKKwl9CisKKwlyZXR1cm4gTlVMTDsKK30KKworLyoKKyAqIENoZWNr IGlmIGEgUENJIGJyaWRnZSBoYXMgaXQncyBBQ1MgcmVkaXJlY3Rpb24gYml0cyBzZXQgdG8gcmVk aXJlY3QgUDJQCisgKiBUTFBzIHVwc3RyZWFtIHZpYSBBQ1MuIFJldHVybnMgMSBpZiB0aGUgcGFj a2V0cyB3aWxsIGJlIHJlZGlyZWN0ZWQKKyAqIHVwc3RyZWFtLCAwIG90aGVyd2lzZS4KKyAqLwor c3RhdGljIGludCBwY2lfYnJpZGdlX2hhc19hY3NfcmVkaXIoc3RydWN0IHBjaV9kZXYgKmRldikK K3sKKwlpbnQgcG9zOworCXUxNiBjdHJsOworCisJcG9zID0gcGNpX2ZpbmRfZXh0X2NhcGFiaWxp dHkoZGV2LCBQQ0lfRVhUX0NBUF9JRF9BQ1MpOworCWlmICghcG9zKQorCQlyZXR1cm4gMDsKKwor CXBjaV9yZWFkX2NvbmZpZ193b3JkKGRldiwgcG9zICsgUENJX0FDU19DVFJMLCAmY3RybCk7CisK KwlpZiAoY3RybCAmIChQQ0lfQUNTX1JSIHwgUENJX0FDU19DUiB8IFBDSV9BQ1NfRUMpKQorCQly ZXR1cm4gMTsKKworCXJldHVybiAwOworfQorCitzdGF0aWMgdm9pZCBzZXFfYnVmX3ByaW50X2J1 c19kZXZmbihzdHJ1Y3Qgc2VxX2J1ZiAqYnVmLCBzdHJ1Y3QgcGNpX2RldiAqZGV2KQoreworCWlm ICghYnVmKQorCQlyZXR1cm47CisKKwlzZXFfYnVmX3ByaW50ZihidWYsICIlMDR4OiUwMng6JTAy eC4leDsiLCBwY2lfZG9tYWluX25yKGRldi0+YnVzKSwKKwkJICAgICAgIGRldi0+YnVzLT5udW1i ZXIsIFBDSV9TTE9UKGRldi0+ZGV2Zm4pLAorCQkgICAgICAgUENJX0ZVTkMoZGV2LT5kZXZmbikp OworfQorCisvKgorICogRmluZCB0aGUgZGlzdGFuY2UgdGhyb3VnaCB0aGUgbmVhcmVzdCBjb21t b24gdXBzdHJlYW0gYnJpZGdlIGJldHdlZW4KKyAqIHR3byBQQ0kgZGV2aWNlcy4KKyAqCisgKiBJ ZiB0aGUgdHdvIGRldmljZXMgYXJlIHRoZSBzYW1lIGRldmljZSB0aGVuIDAgd2lsbCBiZSByZXR1 cm5lZC4KKyAqCisgKiBJZiB0aGVyZSBhcmUgdHdvIHZpcnR1YWwgZnVuY3Rpb25zIG9mIHRoZSBz YW1lIGRldmljZSBiZWhpbmQgdGhlIHNhbWUKKyAqIGJyaWRnZSBwb3J0IHRoZW4gMiB3aWxsIGJl IHJldHVybmVkIChvbmUgc3RlcCBkb3duIHRvIHRoZSBQQ0llIHN3aXRjaCwKKyAqIHRoZW4gb25l IHN0ZXAgYmFjayB0byB0aGUgc2FtZSBkZXZpY2UpLgorICoKKyAqIEluIHRoZSBjYXNlIHdoZXJl IHR3byBkZXZpY2VzIGFyZSBjb25uZWN0ZWQgdG8gdGhlIHNhbWUgUENJZSBzd2l0Y2gsIHRoZQor ICogdmFsdWUgNCB3aWxsIGJlIHJldHVybmVkLiBUaGlzIGNvcnJlc3BvbmRzIHRvIHRoZSBmb2xs b3dpbmcgUENJIHRyZWU6CisgKgorICogICAgIC0rICBSb290IFBvcnQKKyAqICAgICAgXCsgU3dp dGNoIFVwc3RyZWFtIFBvcnQKKyAqICAgICAgICstKyBTd2l0Y2ggRG93bnN0cmVhbSBQb3J0Cisg KiAgICAgICArIFwtIERldmljZSBBCisgKiAgICAgICBcLSsgU3dpdGNoIERvd25zdHJlYW0gUG9y dAorICogICAgICAgICBcLSBEZXZpY2UgQgorICoKKyAqIFRoZSBkaXN0YW5jZSBpcyA0IGJlY2F1 c2Ugd2UgdHJhdmVyc2UgZnJvbSBEZXZpY2UgQSB0aHJvdWdoIHRoZSBkb3duc3RyZWFtCisgKiBw b3J0IG9mIHRoZSBzd2l0Y2gsIHRvIHRoZSBjb21tb24gdXBzdHJlYW0gcG9ydCwgYmFjayB1cCB0 byB0aGUgc2Vjb25kCisgKiBkb3duc3RyZWFtIHBvcnQgYW5kIHRoZW4gdG8gRGV2aWNlIEIuCisg KgorICogQW55IHR3byBkZXZpY2VzIHRoYXQgZG9uJ3QgaGF2ZSBhIGNvbW1vbiB1cHN0cmVhbSBi cmlkZ2Ugd2lsbCByZXR1cm4gLTEuCisgKiBJbiB0aGlzIHdheSBkZXZpY2VzIG9uIHNlcGFyYXRl IFBDSWUgcm9vdCBwb3J0cyB3aWxsIGJlIHJlamVjdGVkLCB3aGljaAorICogaXMgd2hhdCB3ZSB3 YW50IGZvciBwZWVyLXRvLXBlZXIgc2VlaW5nIGVhY2ggUENJZSByb290IHBvcnQgZGVmaW5lcyBh CisgKiBzZXBhcmF0ZSBoaWVyYXJjaHkgZG9tYWluIGFuZCB0aGVyZSdzIG5vIHdheSB0byBkZXRl cm1pbmUgd2hldGhlciB0aGUgcm9vdAorICogY29tcGxleCBzdXBwb3J0cyBmb3J3YXJkaW5nIGJl dHdlZW4gdGhlbS4KKyAqCisgKiBJbiB0aGUgY2FzZSB3aGVyZSB0d28gZGV2aWNlcyBhcmUgY29u bmVjdGVkIHRvIGRpZmZlcmVudCBQQ0llIHN3aXRjaGVzLAorICogdGhpcyBmdW5jdGlvbiB3aWxs IHN0aWxsIHJldHVybiBhIHBvc2l0aXZlIGRpc3RhbmNlIGFzIGxvbmcgYXMgYm90aAorICogc3dp dGNoZXMgZXZlbnV0YWxseSBoYXZlIGEgY29tbW9uIHVwc3RyZWFtIGJyaWRnZS4gTm90ZSB0aGlz IGNvdmVycworICogdGhlIGNhc2Ugb2YgdXNpbmcgbXVsdGlwbGUgUENJZSBzd2l0Y2hlcyB0byBh Y2hpZXZlIGEgZGVzaXJlZCBsZXZlbCBvZgorICogZmFuLW91dCBmcm9tIGEgcm9vdCBwb3J0LiBU aGUgZXhhY3QgZGlzdGFuY2Ugd2lsbCBiZSBhIGZ1bmN0aW9uIG9mIHRoZQorICogbnVtYmVyIG9m IHN3aXRjaGVzIGJldHdlZW4gRGV2aWNlIEEgYW5kIERldmljZSBCLgorICoKKyAqIElmIGEgYnJp ZGdlIHdoaWNoIGhhcyBhbnkgQUNTIHJlZGlyZWN0aW9uIGJpdHMgc2V0IGlzIGluIHRoZSBwYXRo CisgKiB0aGVuIHRoaXMgZnVuY3Rpb25zIHdpbGwgcmV0dXJuIC0yLiBUaGlzIGlzIHNvIHdlIHJl amVjdCBhbnkKKyAqIGNhc2VzIHdoZXJlIHRoZSBUTFBzIGFyZSBmb3J3YXJkZWQgdXAgaW50byB0 aGUgcm9vdCBjb21wbGV4LgorICogSW4gdGhpcyBjYXNlLCBhIGxpc3Qgb2YgYWxsIGluZnJpbmdp bmcgYnJpZGdlIGFkZHJlc3NlcyB3aWxsIGJlCisgKiBwb3B1bGF0ZWQgaW4gYWNzX2xpc3QgKGFz c3VtaW5nIGl0J3Mgbm9uLW51bGwpIGZvciBwcmludGsgcHVycG9zZXMuCisgKi8KK3N0YXRpYyBp bnQgdXBzdHJlYW1fYnJpZGdlX2Rpc3RhbmNlKHN0cnVjdCBwY2lfZGV2ICphLAorCQkJCSAgICBz dHJ1Y3QgcGNpX2RldiAqYiwKKwkJCQkgICAgc3RydWN0IHNlcV9idWYgKmFjc19saXN0KQorewor CWludCBkaXN0X2EgPSAwOworCWludCBkaXN0X2IgPSAwOworCXN0cnVjdCBwY2lfZGV2ICpiYiA9 IE5VTEw7CisJaW50IGFjc19jbnQgPSAwOworCisJLyoKKwkgKiBOb3RlLCB3ZSBkb24ndCBuZWVk IHRvIHRha2UgcmVmZXJlbmNlcyB0byBkZXZpY2VzIHJldHVybmVkIGJ5CisJICogcGNpX3Vwc3Ry ZWFtX2JyaWRnZSgpIHNlZWluZyB3ZSBob2xkIGEgcmVmZXJlbmNlIHRvIGEgY2hpbGQKKwkgKiBk ZXZpY2Ugd2hpY2ggd2lsbCBhbHJlYWR5IGhvbGQgYSByZWZlcmVuY2UgdG8gdGhlIHVwc3RyZWFt IGJyaWRnZS4KKwkgKi8KKworCXdoaWxlIChhKSB7CisJCWRpc3RfYiA9IDA7CisKKwkJaWYgKHBj aV9icmlkZ2VfaGFzX2Fjc19yZWRpcihhKSkgeworCQkJc2VxX2J1Zl9wcmludF9idXNfZGV2Zm4o YWNzX2xpc3QsIGEpOworCQkJYWNzX2NudCsrOworCQl9CisKKwkJYmIgPSBiOworCisJCXdoaWxl IChiYikgeworCQkJaWYgKGEgPT0gYmIpCisJCQkJZ290byBjaGVja19iX3BhdGhfYWNzOworCisJ CQliYiA9IHBjaV91cHN0cmVhbV9icmlkZ2UoYmIpOworCQkJZGlzdF9iKys7CisJCX0KKworCQlh ID0gcGNpX3Vwc3RyZWFtX2JyaWRnZShhKTsKKwkJZGlzdF9hKys7CisJfQorCisJcmV0dXJuIC0x OworCitjaGVja19iX3BhdGhfYWNzOgorCWJiID0gYjsKKworCXdoaWxlIChiYikgeworCQlpZiAo YSA9PSBiYikKKwkJCWJyZWFrOworCisJCWlmIChwY2lfYnJpZGdlX2hhc19hY3NfcmVkaXIoYmIp KSB7CisJCQlzZXFfYnVmX3ByaW50X2J1c19kZXZmbihhY3NfbGlzdCwgYmIpOworCQkJYWNzX2Nu dCsrOworCQl9CisKKwkJYmIgPSBwY2lfdXBzdHJlYW1fYnJpZGdlKGJiKTsKKwl9CisKKwlpZiAo YWNzX2NudCkKKwkJcmV0dXJuIC0yOworCisJcmV0dXJuIGRpc3RfYSArIGRpc3RfYjsKK30KKwor c3RhdGljIGludCB1cHN0cmVhbV9icmlkZ2VfZGlzdGFuY2Vfd2FybihzdHJ1Y3QgcGNpX2RldiAq cHJvdmlkZXIsCisJCQkJCSBzdHJ1Y3QgcGNpX2RldiAqY2xpZW50KQoreworCXN0cnVjdCBzZXFf YnVmIGFjc19saXN0OworCWludCByZXQ7CisKKwlzZXFfYnVmX2luaXQoJmFjc19saXN0LCBrbWFs bG9jKFBBR0VfU0laRSwgR0ZQX0tFUk5FTCksIFBBR0VfU0laRSk7CisKKwlyZXQgPSB1cHN0cmVh bV9icmlkZ2VfZGlzdGFuY2UocHJvdmlkZXIsIGNsaWVudCwgJmFjc19saXN0KTsKKwlpZiAocmV0 ID09IC0yKSB7CisJCXBjaV93YXJuKGNsaWVudCwgImNhbm5vdCBiZSB1c2VkIGZvciBwZWVyLXRv LXBlZXIgRE1BIGFzIEFDUyByZWRpcmVjdCBpcyBzZXQgYmV0d2VlbiB0aGUgY2xpZW50IGFuZCBw cm92aWRlclxuIik7CisJCS8qIERyb3AgZmluYWwgc2VtaWNvbG9uICovCisJCWFjc19saXN0LmJ1 ZmZlclthY3NfbGlzdC5sZW4tMV0gPSAwOworCQlwY2lfd2FybihjbGllbnQsICJ0byBkaXNhYmxl IEFDUyByZWRpcmVjdCBmb3IgdGhpcyBwYXRoLCBhZGQgdGhlIGtlcm5lbCBwYXJhbWV0ZXI6IHBj aT1kaXNhYmxlX2Fjc19yZWRpcj0lc1xuIiwKKwkJCSBhY3NfbGlzdC5idWZmZXIpOworCisJfSBl bHNlIGlmIChyZXQgPCAwKSB7CisJCXBjaV93YXJuKGNsaWVudCwgImNhbm5vdCBiZSB1c2VkIGZv ciBwZWVyLXRvLXBlZXIgRE1BIGFzIHRoZSBjbGllbnQgYW5kIHByb3ZpZGVyIGRvIG5vdCBzaGFy ZSBhbiB1cHN0cmVhbSBicmlkZ2VcbiIpOworCX0KKworCWtmcmVlKGFjc19saXN0LmJ1ZmZlcik7 CisKKwlyZXR1cm4gcmV0OworfQorCitzdHJ1Y3QgcGNpX3AycGRtYV9jbGllbnQgeworCXN0cnVj dCBsaXN0X2hlYWQgbGlzdDsKKwlzdHJ1Y3QgcGNpX2RldiAqY2xpZW50OworCXN0cnVjdCBwY2lf ZGV2ICpwcm92aWRlcjsKK307CisKKy8qKgorICogcGNpX3AycGRtYV9hZGRfY2xpZW50IC0gYWxs b2NhdGUgYSBuZXcgZWxlbWVudCBpbiBhIGNsaWVudCBkZXZpY2UgbGlzdAorICogQGhlYWQ6IGxp c3QgaGVhZCBvZiBwMnBkbWEgY2xpZW50cworICogQGRldjogZGV2aWNlIHRvIGFkZCB0byB0aGUg bGlzdAorICoKKyAqIFRoaXMgYWRkcyBAZGV2IHRvIGEgbGlzdCBvZiBjbGllbnRzIHVzZWQgYnkg YSBwMnBkbWEgZGV2aWNlLgorICogVGhpcyBsaXN0IHNob3VsZCBiZSBwYXNzZWQgdG8gcGNpX3Ay cG1lbV9maW5kKCkuIE9uY2UgcGNpX3AycG1lbV9maW5kKCkgaGFzCisgKiBiZWVuIGNhbGxlZCBz dWNjZXNzZnVsbHksIHRoZSBsaXN0IHdpbGwgYmUgYm91bmQgdG8gYSBzcGVjaWZpYyBwMnBkbWEK KyAqIGRldmljZSBhbmQgbmV3IGNsaWVudHMgY2FuIG9ubHkgYmUgYWRkZWQgdG8gdGhlIGxpc3Qg aWYgdGhleSBhcmUKKyAqIHN1cHBvcnRlZCBieSB0aGF0IHAycGRtYSBkZXZpY2UuCisgKgorICog VGhlIGNhbGxlciBpcyBleHBlY3RlZCB0byBoYXZlIGEgbG9jayB3aGljaCBwcm90ZWN0cyBAaGVh ZCBhcyBuZWNlc3NhcnkKKyAqIHNvIHRoYXQgbm9uZSBvZiB0aGUgcGNpX3AycCBmdW5jdGlvbnMg Y2FuIGJlIGNhbGxlZCBjb25jdXJyZW50bHkKKyAqIG9uIHRoYXQgbGlzdC4KKyAqCisgKiBSZXR1 cm5zIDAgaWYgdGhlIGNsaWVudCB3YXMgc3VjY2Vzc2Z1bGx5IGFkZGVkLgorICovCitpbnQgcGNp X3AycGRtYV9hZGRfY2xpZW50KHN0cnVjdCBsaXN0X2hlYWQgKmhlYWQsIHN0cnVjdCBkZXZpY2Ug KmRldikKK3sKKwlzdHJ1Y3QgcGNpX3AycGRtYV9jbGllbnQgKml0ZW0sICpuZXdfaXRlbTsKKwlz dHJ1Y3QgcGNpX2RldiAqcHJvdmlkZXIgPSBOVUxMOworCXN0cnVjdCBwY2lfZGV2ICpjbGllbnQ7 CisJaW50IHJldDsKKworCWlmIChJU19FTkFCTEVEKENPTkZJR19ETUFfVklSVF9PUFMpICYmIGRl di0+ZG1hX29wcyA9PSAmZG1hX3ZpcnRfb3BzKSB7CisJCWRldl93YXJuKGRldiwgImNhbm5vdCBi ZSB1c2VkIGZvciBwZWVyLXRvLXBlZXIgRE1BIGJlY2F1c2UgdGhlIGRyaXZlciBtYWtlcyB1c2Ug b2YgZG1hX3ZpcnRfb3BzXG4iKTsKKwkJcmV0dXJuIC1FTk9ERVY7CisJfQorCisJY2xpZW50ID0g ZmluZF9wYXJlbnRfcGNpX2RldihkZXYpOworCWlmICghY2xpZW50KSB7CisJCWRldl93YXJuKGRl diwgImNhbm5vdCBiZSB1c2VkIGZvciBwZWVyLXRvLXBlZXIgRE1BIGFzIGl0IGlzIG5vdCBhIFBD SSBkZXZpY2VcbiIpOworCQlyZXR1cm4gLUVOT0RFVjsKKwl9CisKKwlpdGVtID0gbGlzdF9maXJz dF9lbnRyeV9vcl9udWxsKGhlYWQsIHN0cnVjdCBwY2lfcDJwZG1hX2NsaWVudCwgbGlzdCk7CisJ aWYgKGl0ZW0gJiYgaXRlbS0+cHJvdmlkZXIpIHsKKwkJcHJvdmlkZXIgPSBpdGVtLT5wcm92aWRl cjsKKworCQlyZXQgPSB1cHN0cmVhbV9icmlkZ2VfZGlzdGFuY2Vfd2Fybihwcm92aWRlciwgY2xp ZW50KTsKKwkJaWYgKHJldCA8IDApIHsKKwkJCXJldCA9IC1FWERFVjsKKwkJCWdvdG8gcHV0X2Ns aWVudDsKKwkJfQorCX0KKworCW5ld19pdGVtID0ga3phbGxvYyhzaXplb2YoKm5ld19pdGVtKSwg R0ZQX0tFUk5FTCk7CisJaWYgKCFuZXdfaXRlbSkgeworCQlyZXQgPSAtRU5PTUVNOworCQlnb3Rv IHB1dF9jbGllbnQ7CisJfQorCisJbmV3X2l0ZW0tPmNsaWVudCA9IGNsaWVudDsKKwluZXdfaXRl bS0+cHJvdmlkZXIgPSBwY2lfZGV2X2dldChwcm92aWRlcik7CisKKwlsaXN0X2FkZF90YWlsKCZu ZXdfaXRlbS0+bGlzdCwgaGVhZCk7CisKKwlyZXR1cm4gMDsKKworcHV0X2NsaWVudDoKKwlwY2lf ZGV2X3B1dChjbGllbnQpOworCXJldHVybiByZXQ7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lf cDJwZG1hX2FkZF9jbGllbnQpOworCitzdGF0aWMgdm9pZCBwY2lfcDJwZG1hX2NsaWVudF9mcmVl KHN0cnVjdCBwY2lfcDJwZG1hX2NsaWVudCAqaXRlbSkKK3sKKwlsaXN0X2RlbCgmaXRlbS0+bGlz dCk7CisJcGNpX2Rldl9wdXQoaXRlbS0+Y2xpZW50KTsKKwlwY2lfZGV2X3B1dChpdGVtLT5wcm92 aWRlcik7CisJa2ZyZWUoaXRlbSk7Cit9CisKKy8qKgorICogcGNpX3AycGRtYV9yZW1vdmVfY2xp ZW50IC0gcmVtb3ZlIGFuZCBmcmVlIGEgcDJwZG1hIGNsaWVudAorICogQGhlYWQ6IGxpc3QgaGVh ZCBvZiBwMnBkbWEgY2xpZW50cworICogQGRldjogZGV2aWNlIHRvIHJlbW92ZSBmcm9tIHRoZSBs aXN0CisgKgorICogVGhpcyByZW1vdmVzIEBkZXYgZnJvbSBhIGxpc3Qgb2YgY2xpZW50cyB1c2Vk IGJ5IGEgcDJwZG1hIGRldmljZS4KKyAqIFRoZSBjYWxsZXIgaXMgZXhwZWN0ZWQgdG8gaGF2ZSBh IGxvY2sgd2hpY2ggcHJvdGVjdHMgQGhlYWQgYXMgbmVjZXNzYXJ5CisgKiBzbyB0aGF0IG5vbmUg b2YgdGhlIHBjaV9wMnAgZnVuY3Rpb25zIGNhbiBiZSBjYWxsZWQgY29uY3VycmVudGx5CisgKiBv biB0aGF0IGxpc3QuCisgKi8KK3ZvaWQgcGNpX3AycGRtYV9yZW1vdmVfY2xpZW50KHN0cnVjdCBs aXN0X2hlYWQgKmhlYWQsIHN0cnVjdCBkZXZpY2UgKmRldikKK3sKKwlzdHJ1Y3QgcGNpX3AycGRt YV9jbGllbnQgKnBvcywgKnRtcDsKKwlzdHJ1Y3QgcGNpX2RldiAqcGRldjsKKworCXBkZXYgPSBm aW5kX3BhcmVudF9wY2lfZGV2KGRldik7CisJaWYgKCFwZGV2KQorCQlyZXR1cm47CisKKwlsaXN0 X2Zvcl9lYWNoX2VudHJ5X3NhZmUocG9zLCB0bXAsIGhlYWQsIGxpc3QpIHsKKwkJaWYgKHBvcy0+ Y2xpZW50ICE9IHBkZXYpCisJCQljb250aW51ZTsKKworCQlwY2lfcDJwZG1hX2NsaWVudF9mcmVl KHBvcyk7CisJfQorCisJcGNpX2Rldl9wdXQocGRldik7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChw Y2lfcDJwZG1hX3JlbW92ZV9jbGllbnQpOworCisvKioKKyAqIHBjaV9wMnBkbWFfY2xpZW50X2xp c3RfZnJlZSAtIGZyZWUgYW4gZW50aXJlIGxpc3Qgb2YgcDJwZG1hIGNsaWVudHMKKyAqIEBoZWFk OiBsaXN0IGhlYWQgb2YgcDJwZG1hIGNsaWVudHMKKyAqCisgKiBUaGlzIHJlbW92ZXMgYWxsIGRl dmljZXMgaW4gYSBsaXN0IG9mIGNsaWVudHMgdXNlZCBieSBhIHAycGRtYSBkZXZpY2UuCisgKiBU aGUgY2FsbGVyIGlzIGV4cGVjdGVkIHRvIGhhdmUgYSBsb2NrIHdoaWNoIHByb3RlY3RzIEBoZWFk IGFzIG5lY2Vzc2FyeQorICogc28gdGhhdCBub25lIG9mIHRoZSBwY2lfcDJwZG1hIGZ1bmN0aW9u cyBjYW4gYmUgY2FsbGVkIGNvbmN1cnJlbnRseQorICogb24gdGhhdCBsaXN0LgorICovCit2b2lk IHBjaV9wMnBkbWFfY2xpZW50X2xpc3RfZnJlZShzdHJ1Y3QgbGlzdF9oZWFkICpoZWFkKQorewor CXN0cnVjdCBwY2lfcDJwZG1hX2NsaWVudCAqcG9zLCAqdG1wOworCisJbGlzdF9mb3JfZWFjaF9l bnRyeV9zYWZlKHBvcywgdG1wLCBoZWFkLCBsaXN0KQorCQlwY2lfcDJwZG1hX2NsaWVudF9mcmVl KHBvcyk7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lfcDJwZG1hX2NsaWVudF9saXN0X2ZyZWUp OworCisvKioKKyAqIHBjaV9wMnBkbWFfZGlzdGFuY2UgLSBEZXRlcm1pdmUgdGhlIGN1bXVsYXRp dmUgZGlzdGFuY2UgYmV0d2VlbgorICoJYSBwMnBkbWEgcHJvdmlkZXIgYW5kIHRoZSBjbGllbnRz IGluIHVzZS4KKyAqIEBwcm92aWRlcjogcDJwZG1hIHByb3ZpZGVyIHRvIGNoZWNrIGFnYWluc3Qg dGhlIGNsaWVudCBsaXN0CisgKiBAY2xpZW50czogbGlzdCBvZiBkZXZpY2VzIHRvIGNoZWNrIChO VUxMLXRlcm1pbmF0ZWQpCisgKiBAdmVyYm9zZTogaWYgdHJ1ZSwgcHJpbnQgd2FybmluZ3MgZm9y IGRldmljZXMgd2hlbiB3ZSByZXR1cm4gLTEKKyAqCisgKiBSZXR1cm5zIC0xIGlmIGFueSBvZiB0 aGUgY2xpZW50cyBhcmUgbm90IGNvbXBhdGlibGUgKGJlaGluZCB0aGUgc2FtZQorICogcm9vdCBw b3J0IGFzIHRoZSBwcm92aWRlciksIG90aGVyd2lzZSByZXR1cm5zIGEgcG9zaXRpdmUgbnVtYmVy IHdoZXJlCisgKiB0aGUgbG93ZXIgbnVtYmVyIGlzIHRoZSBwcmVmZXJyYWJsZSBjaG9pY2UuIChJ ZiB0aGVyZSdzIG9uZSBjbGllbnQKKyAqIHRoYXQncyB0aGUgc2FtZSBhcyB0aGUgcHJvdmlkZXIg aXQgd2lsbCByZXR1cm4gMCwgd2hpY2ggaXMgYmVzdCBjaG9pY2UpLgorICoKKyAqIEZvciBub3cs ICJjb21wYXRpYmxlIiBtZWFucyB0aGUgcHJvdmlkZXIgYW5kIHRoZSBjbGllbnRzIGFyZSBhbGwg YmVoaW5kCisgKiB0aGUgc2FtZSBQQ0kgcm9vdCBwb3J0LiBUaGlzIGN1dHMgb3V0IGNhc2VzIHRo YXQgbWF5IHdvcmsgYnV0IGlzIHNhZmVzdAorICogZm9yIHRoZSB1c2VyLiBGdXR1cmUgd29yayBj YW4gZXhwYW5kIHRoaXMgdG8gd2hpdGUtbGlzdCByb290IGNvbXBsZXhlcyB0aGF0CisgKiBjYW4g c2FmZWx5IGZvcndhcmQgYmV0d2VlbiBlYWNoIHBvcnRzLgorICovCitpbnQgcGNpX3AycGRtYV9k aXN0YW5jZShzdHJ1Y3QgcGNpX2RldiAqcHJvdmlkZXIsIHN0cnVjdCBsaXN0X2hlYWQgKmNsaWVu dHMsCisJCQlib29sIHZlcmJvc2UpCit7CisJc3RydWN0IHBjaV9wMnBkbWFfY2xpZW50ICpwb3M7 CisJaW50IHJldDsKKwlpbnQgZGlzdGFuY2UgPSAwOworCWJvb2wgbm90X3N1cHBvcnRlZCA9IGZh bHNlOworCisJaWYgKGxpc3RfZW1wdHkoY2xpZW50cykpCisJCXJldHVybiAtMTsKKworCWxpc3Rf Zm9yX2VhY2hfZW50cnkocG9zLCBjbGllbnRzLCBsaXN0KSB7CisJCWlmICh2ZXJib3NlKQorCQkJ cmV0ID0gdXBzdHJlYW1fYnJpZGdlX2Rpc3RhbmNlX3dhcm4ocHJvdmlkZXIsCisJCQkJCQkJICAg IHBvcy0+Y2xpZW50KTsKKwkJZWxzZQorCQkJcmV0ID0gdXBzdHJlYW1fYnJpZGdlX2Rpc3RhbmNl KHByb3ZpZGVyLCBwb3MtPmNsaWVudCwKKwkJCQkJCSAgICAgICBOVUxMKTsKKworCQlpZiAocmV0 IDwgMCkKKwkJCW5vdF9zdXBwb3J0ZWQgPSB0cnVlOworCisJCWlmIChub3Rfc3VwcG9ydGVkICYm ICF2ZXJib3NlKQorCQkJYnJlYWs7CisKKwkJZGlzdGFuY2UgKz0gcmV0OworCX0KKworCWlmIChu b3Rfc3VwcG9ydGVkKQorCQlyZXR1cm4gLTE7CisKKwlyZXR1cm4gZGlzdGFuY2U7Cit9CitFWFBP UlRfU1lNQk9MX0dQTChwY2lfcDJwZG1hX2Rpc3RhbmNlKTsKKworLyoqCisgKiBwY2lfcDJwZG1h X2Fzc2lnbl9wcm92aWRlciAtIENoZWNrIGNvbXBhdGliaWx5IChhcyBwZXIgcGNpX3AycGRtYV9k aXN0YW5jZSkKKyAqCWFuZCBhc3NpZ24gYSBwcm92aWRlciB0byBhIGxpc3Qgb2YgY2xpZW50cwor ICogQHByb3ZpZGVyOiBwMnBkbWEgcHJvdmlkZXIgdG8gYXNzaWduIHRvIHRoZSBjbGllbnQgbGlz dAorICogQGNsaWVudHM6IGxpc3Qgb2YgZGV2aWNlcyB0byBjaGVjayAoTlVMTC10ZXJtaW5hdGVk KQorICoKKyAqIFJldHVybnMgZmFsc2UgaWYgYW55IG9mIHRoZSBjbGllbnRzIGFyZSBub3QgY29t cGF0aWJsZSwgdHJ1ZSBpZiB0aGUKKyAqIHByb3ZpZGVyIHdhcyBzdWNjZXNzZnVsbHkgYXNzaWdu ZWQgdG8gdGhlIGNsaWVudHMuCisgKi8KK2Jvb2wgcGNpX3AycGRtYV9hc3NpZ25fcHJvdmlkZXIo c3RydWN0IHBjaV9kZXYgKnByb3ZpZGVyLAorCQkJCXN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMp Cit7CisJc3RydWN0IHBjaV9wMnBkbWFfY2xpZW50ICpwb3M7CisKKwlpZiAocGNpX3AycGRtYV9k aXN0YW5jZShwcm92aWRlciwgY2xpZW50cywgdHJ1ZSkgPCAwKQorCQlyZXR1cm4gZmFsc2U7CisK KwlsaXN0X2Zvcl9lYWNoX2VudHJ5KHBvcywgY2xpZW50cywgbGlzdCkKKwkJcG9zLT5wcm92aWRl ciA9IHByb3ZpZGVyOworCisJcmV0dXJuIHRydWU7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lf cDJwZG1hX2Fzc2lnbl9wcm92aWRlcik7CisKKy8qKgorICogcGNpX2hhc19wMnBtZW0gLSBjaGVj ayBpZiBhIGdpdmVuIFBDSSBkZXZpY2UgaGFzIHB1Ymxpc2hlZCBhbnkgcDJwbWVtCisgKiBAcGRl djogUENJIGRldmljZSB0byBjaGVjaworICovCitib29sIHBjaV9oYXNfcDJwbWVtKHN0cnVjdCBw Y2lfZGV2ICpwZGV2KQoreworCXJldHVybiBwZGV2LT5wMnBkbWEgJiYgcGRldi0+cDJwZG1hLT5w MnBtZW1fcHVibGlzaGVkOworfQorRVhQT1JUX1NZTUJPTF9HUEwocGNpX2hhc19wMnBtZW0pOwor CisvKioKKyAqIHBjaV9wMnBtZW1fZmluZCAtIGZpbmQgYSBwZWVyLXRvLXBlZXIgRE1BIG1lbW9y eSBkZXZpY2UgY29tcGF0aWJsZSB3aXRoCisgKgl0aGUgc3BlY2lmaWVkIGxpc3Qgb2YgY2xpZW50 cyBhbmQgc2hvcnRlc3QgZGlzdGFuY2UgKGFzIGRldGVybWluZWQKKyAqCWJ5IHBjaV9wMnBtZW1f ZG1hKCkpCisgKiBAY2xpZW50czogbGlzdCBvZiBkZXZpY2VzIHRvIGNoZWNrIChOVUxMLXRlcm1p bmF0ZWQpCisgKgorICogSWYgbXVsdGlwbGUgZGV2aWNlcyBhcmUgYmVoaW5kIHRoZSBzYW1lIHN3 aXRjaCwgdGhlIG9uZSAiY2xvc2VzdCIgdG8gdGhlCisgKiBjbGllbnQgZGV2aWNlcyBpbiB1c2Ug d2lsbCBiZSBjaG9zZW4gZmlyc3QuIChTbyBpZiBvbmUgb2YgdGhlIHByb3ZpZGVycyBhcmUKKyAq IHRoZSBzYW1lIGFzIG9uZSBvZiB0aGUgY2xpZW50cywgdGhhdCBwcm92aWRlciB3aWxsIGJlIHVz ZWQgYWhlYWQgb2YgYW55CisgKiBvdGhlciBwcm92aWRlcnMgdGhhdCBhcmUgdW5yZWxhdGVkKS4g SWYgbXVsdGlwbGUgcHJvdmlkZXJzIGFyZSBhbiBlcXVhbAorICogZGlzdGFuY2UgYXdheSwgb25l IHdpbGwgYmUgY2hvc2VuIGF0IHJhbmRvbS4KKyAqCisgKiBSZXR1cm5zIGEgcG9pbnRlciB0byB0 aGUgUENJIGRldmljZSB3aXRoIGEgcmVmZXJlbmNlIHRha2VuICh1c2UgcGNpX2Rldl9wdXQKKyAq IHRvIHJldHVybiB0aGUgcmVmZXJlbmNlKSBvciBOVUxMIGlmIG5vIGNvbXBhdGlibGUgZGV2aWNl IGlzIGZvdW5kLiBUaGUKKyAqIGZvdW5kIHByb3ZpZGVyIHdpbGwgYWxzbyBiZSBhc3NpZ25lZCB0 byB0aGUgY2xpZW50IGxpc3QuCisgKi8KK3N0cnVjdCBwY2lfZGV2ICpwY2lfcDJwbWVtX2ZpbmQo c3RydWN0IGxpc3RfaGVhZCAqY2xpZW50cykKK3sKKwlzdHJ1Y3QgcGNpX2RldiAqcGRldiA9IE5V TEw7CisJc3RydWN0IHBjaV9wMnBkbWFfY2xpZW50ICpwb3M7CisJaW50IGRpc3RhbmNlOworCWlu dCBjbG9zZXN0X2Rpc3RhbmNlID0gSU5UX01BWDsKKwlzdHJ1Y3QgcGNpX2RldiAqKmNsb3Nlc3Rf cGRldnM7CisJaW50IGRldl9jbnQgPSAwOworCWNvbnN0IGludCBtYXhfZGV2cyA9IFBBR0VfU0la RSAvIHNpemVvZigqY2xvc2VzdF9wZGV2cyk7CisJaW50IGk7CisKKwljbG9zZXN0X3BkZXZzID0g a21hbGxvYyhQQUdFX1NJWkUsIEdGUF9LRVJORUwpOworCisJd2hpbGUgKChwZGV2ID0gcGNpX2dl dF9kZXZpY2UoUENJX0FOWV9JRCwgUENJX0FOWV9JRCwgcGRldikpKSB7CisJCWlmICghcGNpX2hh c19wMnBtZW0ocGRldikpCisJCQljb250aW51ZTsKKworCQlkaXN0YW5jZSA9IHBjaV9wMnBkbWFf ZGlzdGFuY2UocGRldiwgY2xpZW50cywgZmFsc2UpOworCQlpZiAoZGlzdGFuY2UgPCAwIHx8IGRp c3RhbmNlID4gY2xvc2VzdF9kaXN0YW5jZSkKKwkJCWNvbnRpbnVlOworCisJCWlmIChkaXN0YW5j ZSA9PSBjbG9zZXN0X2Rpc3RhbmNlICYmIGRldl9jbnQgPj0gbWF4X2RldnMpCisJCQljb250aW51 ZTsKKworCQlpZiAoZGlzdGFuY2UgPCBjbG9zZXN0X2Rpc3RhbmNlKSB7CisJCQlmb3IgKGkgPSAw OyBpIDwgZGV2X2NudDsgaSsrKQorCQkJCXBjaV9kZXZfcHV0KGNsb3Nlc3RfcGRldnNbaV0pOwor CisJCQlkZXZfY250ID0gMDsKKwkJCWNsb3Nlc3RfZGlzdGFuY2UgPSBkaXN0YW5jZTsKKwkJfQor CisJCWNsb3Nlc3RfcGRldnNbZGV2X2NudCsrXSA9IHBjaV9kZXZfZ2V0KHBkZXYpOworCX0KKwor CWlmIChkZXZfY250KQorCQlwZGV2ID0gcGNpX2Rldl9nZXQoY2xvc2VzdF9wZGV2c1twcmFuZG9t X3UzMl9tYXgoZGV2X2NudCldKTsKKworCWZvciAoaSA9IDA7IGkgPCBkZXZfY250OyBpKyspCisJ CXBjaV9kZXZfcHV0KGNsb3Nlc3RfcGRldnNbaV0pOworCisJaWYgKHBkZXYpCisJCWxpc3RfZm9y X2VhY2hfZW50cnkocG9zLCBjbGllbnRzLCBsaXN0KQorCQkJcG9zLT5wcm92aWRlciA9IHBkZXY7 CisKKwlrZnJlZShjbG9zZXN0X3BkZXZzKTsKKwlyZXR1cm4gcGRldjsKK30KK0VYUE9SVF9TWU1C T0xfR1BMKHBjaV9wMnBtZW1fZmluZCk7CisKKy8qKgorICogcGNpX2FsbG9jX3AycF9tZW0gLSBh bGxvY2F0ZSBwZWVyLXRvLXBlZXIgRE1BIG1lbW9yeQorICogQHBkZXY6IHRoZSBkZXZpY2UgdG8g YWxsb2NhdGUgbWVtb3J5IGZyb20KKyAqIEBzaXplOiBudW1iZXIgb2YgYnl0ZXMgdG8gYWxsb2Nh dGUKKyAqCisgKiBSZXR1cm5zIHRoZSBhbGxvY2F0ZWQgbWVtb3J5IG9yIE5VTEwgb24gZXJyb3Iu CisgKi8KK3ZvaWQgKnBjaV9hbGxvY19wMnBtZW0oc3RydWN0IHBjaV9kZXYgKnBkZXYsIHNpemVf dCBzaXplKQoreworCXZvaWQgKnJldDsKKworCWlmICh1bmxpa2VseSghcGRldi0+cDJwZG1hKSkK KwkJcmV0dXJuIE5VTEw7CisKKwlpZiAodW5saWtlbHkoIXBlcmNwdV9yZWZfdHJ5Z2V0X2xpdmUo JnBkZXYtPnAycGRtYS0+ZGV2bWFwX3JlZikpKQorCQlyZXR1cm4gTlVMTDsKKworCXJldCA9ICh2 b2lkICopZ2VuX3Bvb2xfYWxsb2MocGRldi0+cDJwZG1hLT5wb29sLCBzaXplKTsKKworCWlmICh1 bmxpa2VseSghcmV0KSkKKwkJcGVyY3B1X3JlZl9wdXQoJnBkZXYtPnAycGRtYS0+ZGV2bWFwX3Jl Zik7CisKKwlyZXR1cm4gcmV0OworfQorRVhQT1JUX1NZTUJPTF9HUEwocGNpX2FsbG9jX3AycG1l bSk7CisKKy8qKgorICogcGNpX2ZyZWVfcDJwbWVtIC0gYWxsb2NhdGUgcGVlci10by1wZWVyIERN QSBtZW1vcnkKKyAqIEBwZGV2OiB0aGUgZGV2aWNlIHRoZSBtZW1vcnkgd2FzIGFsbG9jYXRlZCBm cm9tCisgKiBAYWRkcjogYWRkcmVzcyBvZiB0aGUgbWVtb3J5IHRoYXQgd2FzIGFsbG9jYXRlZAor ICogQHNpemU6IG51bWJlciBvZiBieXRlcyB0aGF0IHdhcyBhbGxvY2F0ZWQKKyAqLwordm9pZCBw Y2lfZnJlZV9wMnBtZW0oc3RydWN0IHBjaV9kZXYgKnBkZXYsIHZvaWQgKmFkZHIsIHNpemVfdCBz aXplKQoreworCWdlbl9wb29sX2ZyZWUocGRldi0+cDJwZG1hLT5wb29sLCAodWludHB0cl90KWFk ZHIsIHNpemUpOworCXBlcmNwdV9yZWZfcHV0KCZwZGV2LT5wMnBkbWEtPmRldm1hcF9yZWYpOwor fQorRVhQT1JUX1NZTUJPTF9HUEwocGNpX2ZyZWVfcDJwbWVtKTsKKworLyoqCisgKiBwY2lfdmly dF90b19idXMgLSByZXR1cm4gdGhlIFBDSSBidXMgYWRkcmVzcyBmb3IgYSBnaXZlbiB2aXJ0dWFs CisgKglhZGRyZXNzIG9idGFpbmVkIHdpdGggcGNpX2FsbG9jX3AycG1lbSgpCisgKiBAcGRldjog dGhlIGRldmljZSB0aGUgbWVtb3J5IHdhcyBhbGxvY2F0ZWQgZnJvbQorICogQGFkZHI6IGFkZHJl c3Mgb2YgdGhlIG1lbW9yeSB0aGF0IHdhcyBhbGxvY2F0ZWQKKyAqLworcGNpX2J1c19hZGRyX3Qg cGNpX3AycG1lbV92aXJ0X3RvX2J1cyhzdHJ1Y3QgcGNpX2RldiAqcGRldiwgdm9pZCAqYWRkcikK K3sKKwlpZiAoIWFkZHIpCisJCXJldHVybiAwOworCWlmICghcGRldi0+cDJwZG1hKQorCQlyZXR1 cm4gMDsKKworCS8qCisJICogTm90ZTogd2hlbiB3ZSBhZGRlZCB0aGUgbWVtb3J5IHRvIHRoZSBw b29sIHdlIHVzZWQgdGhlIFBDSQorCSAqIGJ1cyBhZGRyZXNzIGFzIHRoZSBwaHlzaWNhbCBhZGRy ZXNzLiBTbyBnZW5fcG9vbF92aXJ0X3RvX3BoeXMoKQorCSAqIGFjdHVhbGx5IHJldHVybnMgdGhl IGJ1cyBhZGRyZXNzIGRlc3BpdGUgdGhlIG1pc2xlYWRpbmcgbmFtZS4KKwkgKi8KKwlyZXR1cm4g Z2VuX3Bvb2xfdmlydF90b19waHlzKHBkZXYtPnAycGRtYS0+cG9vbCwgKHVuc2lnbmVkIGxvbmcp YWRkcik7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lfcDJwbWVtX3ZpcnRfdG9fYnVzKTsKKwor LyoqCisgKiBwY2lfcDJwbWVtX2FsbG9jX3NnbCAtIGFsbG9jYXRlIHBlZXItdG8tcGVlciBETUEg bWVtb3J5IGluIGEgc2NhdHRlcmxpc3QKKyAqIEBwZGV2OiB0aGUgZGV2aWNlIHRvIGFsbG9jYXRl IG1lbW9yeSBmcm9tCisgKiBAc2dsOiB0aGUgYWxsb2NhdGVkIHNjYXR0ZXJsaXN0CisgKiBAbmVu dHM6IHRoZSBudW1iZXIgb2YgU0cgZW50cmllcyBpbiB0aGUgbGlzdAorICogQGxlbmd0aDogbnVt YmVyIG9mIGJ5dGVzIHRvIGFsbG9jYXRlCisgKgorICogUmV0dXJucyAwIG9uIHN1Y2Nlc3MKKyAq Lworc3RydWN0IHNjYXR0ZXJsaXN0ICpwY2lfcDJwbWVtX2FsbG9jX3NnbChzdHJ1Y3QgcGNpX2Rl diAqcGRldiwKKwkJCQkJIHVuc2lnbmVkIGludCAqbmVudHMsIHUzMiBsZW5ndGgpCit7CisJc3Ry dWN0IHNjYXR0ZXJsaXN0ICpzZzsKKwl2b2lkICphZGRyOworCisJc2cgPSBremFsbG9jKHNpemVv Zigqc2cpLCBHRlBfS0VSTkVMKTsKKwlpZiAoIXNnKQorCQlyZXR1cm4gTlVMTDsKKworCXNnX2lu aXRfdGFibGUoc2csIDEpOworCisJYWRkciA9IHBjaV9hbGxvY19wMnBtZW0ocGRldiwgbGVuZ3Ro KTsKKwlpZiAoIWFkZHIpCisJCWdvdG8gb3V0X2ZyZWVfc2c7CisKKwlzZ19zZXRfYnVmKHNnLCBh ZGRyLCBsZW5ndGgpOworCSpuZW50cyA9IDE7CisJcmV0dXJuIHNnOworCitvdXRfZnJlZV9zZzoK KwlrZnJlZShzZyk7CisJcmV0dXJuIE5VTEw7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lfcDJw bWVtX2FsbG9jX3NnbCk7CisKKy8qKgorICogcGNpX3AycG1lbV9mcmVlX3NnbCAtIGZyZWUgYSBz Y2F0dGVybGlzdCBhbGxvY2F0ZWQgYnkgcGNpX3AycG1lbV9hbGxvY19zZ2woKQorICogQHBkZXY6 IHRoZSBkZXZpY2UgdG8gYWxsb2NhdGUgbWVtb3J5IGZyb20KKyAqIEBzZ2w6IHRoZSBhbGxvY2F0 ZWQgc2NhdHRlcmxpc3QKKyAqIEBuZW50czogdGhlIG51bWJlciBvZiBTRyBlbnRyaWVzIGluIHRo ZSBsaXN0CisgKi8KK3ZvaWQgcGNpX3AycG1lbV9mcmVlX3NnbChzdHJ1Y3QgcGNpX2RldiAqcGRl diwgc3RydWN0IHNjYXR0ZXJsaXN0ICpzZ2wpCit7CisJc3RydWN0IHNjYXR0ZXJsaXN0ICpzZzsK KwlpbnQgY291bnQ7CisKKwlmb3JfZWFjaF9zZyhzZ2wsIHNnLCBJTlRfTUFYLCBjb3VudCkgewor CQlpZiAoIXNnKQorCQkJYnJlYWs7CisKKwkJcGNpX2ZyZWVfcDJwbWVtKHBkZXYsIHNnX3ZpcnQo c2cpLCBzZy0+bGVuZ3RoKTsKKwl9CisJa2ZyZWUoc2dsKTsKK30KK0VYUE9SVF9TWU1CT0xfR1BM KHBjaV9wMnBtZW1fZnJlZV9zZ2wpOworCisvKioKKyAqIHBjaV9wMnBtZW1fcHVibGlzaCAtIHB1 Ymxpc2ggdGhlIHBlZXItdG8tcGVlciBETUEgbWVtb3J5IGZvciB1c2UgYnkKKyAqCW90aGVyIGRl dmljZXMgd2l0aCBwY2lfcDJwbWVtX2ZpbmQoKQorICogQHBkZXY6IHRoZSBkZXZpY2Ugd2l0aCBw ZWVyLXRvLXBlZXIgRE1BIG1lbW9yeSB0byBwdWJsaXNoCisgKiBAcHVibGlzaDogc2V0IHRvIHRy dWUgdG8gcHVibGlzaCB0aGUgbWVtb3J5LCBmYWxzZSB0byB1bnB1Ymxpc2ggaXQKKyAqCisgKiBQ dWJsaXNoZWQgbWVtb3J5IGNhbiBiZSB1c2VkIGJ5IG90aGVyIFBDSSBkZXZpY2UgZHJpdmVycyBm b3IKKyAqIHBlZXItMi1wZWVyIERNQSBvcGVyYXRpb25zLiBOb24tcHVibGlzaGVkIG1lbW9yeSBp cyByZXNlcnZlZCBmb3IKKyAqIGV4bHVzaXZlIHVzZSBvZiB0aGUgZGV2aWNlIGRyaXZlciB0aGF0 IHJlZ2lzdGVycyB0aGUgcGVlci10by1wZWVyCisgKiBtZW1vcnkuCisgKi8KK3ZvaWQgcGNpX3Ay cG1lbV9wdWJsaXNoKHN0cnVjdCBwY2lfZGV2ICpwZGV2LCBib29sIHB1Ymxpc2gpCit7CisJaWYg KHB1Ymxpc2ggJiYgIXBkZXYtPnAycGRtYSkKKwkJcmV0dXJuOworCisJcGRldi0+cDJwZG1hLT5w MnBtZW1fcHVibGlzaGVkID0gcHVibGlzaDsKK30KK0VYUE9SVF9TWU1CT0xfR1BMKHBjaV9wMnBt ZW1fcHVibGlzaCk7CmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21lbXJlbWFwLmggYi9pbmNs dWRlL2xpbnV4L21lbXJlbWFwLmgKaW5kZXggZjkxZjllNzYzNTU3Li45NTUzMzcwZWJkYWQgMTAw NjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgvbWVtcmVtYXAuaAorKysgYi9pbmNsdWRlL2xpbnV4L21l bXJlbWFwLmgKQEAgLTUzLDExICs1MywxNiBAQCBzdHJ1Y3Qgdm1lbV9hbHRtYXAgewogICogd2Fr ZXVwIGV2ZW50IHdoZW5ldmVyIGEgcGFnZSBpcyB1bnBpbm5lZCBhbmQgYmVjb21lcyBpZGxlLiBU aGlzCiAgKiB3YWtldXAgaXMgdXNlZCB0byBjb29yZGluYXRlIHBoeXNpY2FsIGFkZHJlc3Mgc3Bh Y2UgbWFuYWdlbWVudCAoZXg6CiAgKiBmcyB0cnVuY2F0ZS9ob2xlIHB1bmNoKSB2cyBwaW5uZWQg cGFnZXMgKGV4OiBkZXZpY2UgZG1hKS4KKyAqCisgKiBNRU1PUllfREVWSUNFX1BDSV9QMlBETUE6 CisgKiBEZXZpY2UgbWVtb3J5IHJlc2lkaW5nIGluIGEgUENJIEJBUiBpbnRlbmRlZCBmb3IgdXNl IHdpdGggUGVlci10by1QZWVyCisgKiB0cmFuc2FjdGlvbnMuCiAgKi8KIGVudW0gbWVtb3J5X3R5 cGUgewogCU1FTU9SWV9ERVZJQ0VfUFJJVkFURSA9IDEsCiAJTUVNT1JZX0RFVklDRV9QVUJMSUMs CiAJTUVNT1JZX0RFVklDRV9GU19EQVgsCisJTUVNT1JZX0RFVklDRV9QQ0lfUDJQRE1BLAogfTsK IAogLyoKZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvbW0uaCBiL2luY2x1ZGUvbGludXgvbW0u aAppbmRleCBhNjFlYmU4YWQ0Y2EuLjIwNTVkZjQxMmE3NyAxMDA2NDQKLS0tIGEvaW5jbHVkZS9s aW51eC9tbS5oCisrKyBiL2luY2x1ZGUvbGludXgvbW0uaApAQCAtODkwLDYgKzg5MCwxOSBAQCBz dGF0aWMgaW5saW5lIGJvb2wgaXNfZGV2aWNlX3B1YmxpY19wYWdlKGNvbnN0IHN0cnVjdCBwYWdl ICpwYWdlKQogCQlwYWdlLT5wZ21hcC0+dHlwZSA9PSBNRU1PUllfREVWSUNFX1BVQkxJQzsKIH0K IAorI2lmZGVmIENPTkZJR19QQ0lfUDJQRE1BCitzdGF0aWMgaW5saW5lIGJvb2wgaXNfcGNpX3Ay cGRtYV9wYWdlKGNvbnN0IHN0cnVjdCBwYWdlICpwYWdlKQoreworCXJldHVybiBpc196b25lX2Rl dmljZV9wYWdlKHBhZ2UpICYmCisJCXBhZ2UtPnBnbWFwLT50eXBlID09IE1FTU9SWV9ERVZJQ0Vf UENJX1AyUERNQTsKK30KKyNlbHNlIC8qIENPTkZJR19QQ0lfUDJQRE1BICovCitzdGF0aWMgaW5s aW5lIGJvb2wgaXNfcGNpX3AycGRtYV9wYWdlKGNvbnN0IHN0cnVjdCBwYWdlICpwYWdlKQorewor CXJldHVybiBmYWxzZTsKK30KKyNlbmRpZiAvKiBDT05GSUdfUENJX1AyUERNQSAqLworCiAjZWxz ZSAvKiBDT05GSUdfREVWX1BBR0VNQVBfT1BTICovCiBzdGF0aWMgaW5saW5lIHZvaWQgZGV2X3Bh Z2VtYXBfZ2V0X29wcyh2b2lkKQogewpAQCAtOTEzLDYgKzkyNiwxMSBAQCBzdGF0aWMgaW5saW5l IGJvb2wgaXNfZGV2aWNlX3B1YmxpY19wYWdlKGNvbnN0IHN0cnVjdCBwYWdlICpwYWdlKQogewog CXJldHVybiBmYWxzZTsKIH0KKworc3RhdGljIGlubGluZSBib29sIGlzX3BjaV9wMnBkbWFfcGFn ZShjb25zdCBzdHJ1Y3QgcGFnZSAqcGFnZSkKK3sKKwlyZXR1cm4gZmFsc2U7Cit9CiAjZW5kaWYg LyogQ09ORklHX0RFVl9QQUdFTUFQX09QUyAqLwogCiBzdGF0aWMgaW5saW5lIHZvaWQgZ2V0X3Bh Z2Uoc3RydWN0IHBhZ2UgKnBhZ2UpCmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L3BjaS1wMnBk bWEuaCBiL2luY2x1ZGUvbGludXgvcGNpLXAycGRtYS5oCm5ldyBmaWxlIG1vZGUgMTAwNjQ0Cmlu ZGV4IDAwMDAwMDAwMDAwMC4uN2IyYjBmNTQ3NTI4Ci0tLSAvZGV2L251bGwKKysrIGIvaW5jbHVk ZS9saW51eC9wY2ktcDJwZG1hLmgKQEAgLTAsMCArMSwxMDIgQEAKKy8qIFNQRFgtTGljZW5zZS1J ZGVudGlmaWVyOiBHUEwtMi4wICovCisvKgorICogUENJIFBlZXIgMiBQZWVyIERNQSBzdXBwb3J0 LgorICoKKyAqIENvcHlyaWdodCAoYykgMjAxNi0yMDE4LCBMb2dhbiBHdW50aG9ycGUKKyAqIENv cHlyaWdodCAoYykgMjAxNi0yMDE3LCBNaWNyb3NlbWkgQ29ycG9yYXRpb24KKyAqIENvcHlyaWdo dCAoYykgMjAxNywgQ2hyaXN0b3BoIEhlbGx3aWcKKyAqIENvcHlyaWdodCAoYykgMjAxOCwgRWlk ZXRpY29tIEluYy4KKyAqCisgKi8KKworI2lmbmRlZiBfTElOVVhfUENJX1AyUERNQV9ICisjZGVm aW5lIF9MSU5VWF9QQ0lfUDJQRE1BX0gKKworI2luY2x1ZGUgPGxpbnV4L3BjaS5oPgorCitzdHJ1 Y3QgYmxvY2tfZGV2aWNlOworc3RydWN0IHNjYXR0ZXJsaXN0OworCisjaWZkZWYgQ09ORklHX1BD SV9QMlBETUEKK2ludCBwY2lfcDJwZG1hX2FkZF9yZXNvdXJjZShzdHJ1Y3QgcGNpX2RldiAqcGRl diwgaW50IGJhciwgc2l6ZV90IHNpemUsCisJCXU2NCBvZmZzZXQpOworaW50IHBjaV9wMnBkbWFf YWRkX2NsaWVudChzdHJ1Y3QgbGlzdF9oZWFkICpoZWFkLCBzdHJ1Y3QgZGV2aWNlICpkZXYpOwor dm9pZCBwY2lfcDJwZG1hX3JlbW92ZV9jbGllbnQoc3RydWN0IGxpc3RfaGVhZCAqaGVhZCwgc3Ry dWN0IGRldmljZSAqZGV2KTsKK3ZvaWQgcGNpX3AycGRtYV9jbGllbnRfbGlzdF9mcmVlKHN0cnVj dCBsaXN0X2hlYWQgKmhlYWQpOworaW50IHBjaV9wMnBkbWFfZGlzdGFuY2Uoc3RydWN0IHBjaV9k ZXYgKnByb3ZpZGVyLCBzdHJ1Y3QgbGlzdF9oZWFkICpjbGllbnRzLAorCQkJYm9vbCB2ZXJib3Nl KTsKK2Jvb2wgcGNpX3AycGRtYV9hc3NpZ25fcHJvdmlkZXIoc3RydWN0IHBjaV9kZXYgKnByb3Zp ZGVyLAorCQkJCXN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMpOworYm9vbCBwY2lfaGFzX3AycG1l bShzdHJ1Y3QgcGNpX2RldiAqcGRldik7CitzdHJ1Y3QgcGNpX2RldiAqcGNpX3AycG1lbV9maW5k KHN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMpOwordm9pZCAqcGNpX2FsbG9jX3AycG1lbShzdHJ1 Y3QgcGNpX2RldiAqcGRldiwgc2l6ZV90IHNpemUpOwordm9pZCBwY2lfZnJlZV9wMnBtZW0oc3Ry dWN0IHBjaV9kZXYgKnBkZXYsIHZvaWQgKmFkZHIsIHNpemVfdCBzaXplKTsKK3BjaV9idXNfYWRk cl90IHBjaV9wMnBtZW1fdmlydF90b19idXMoc3RydWN0IHBjaV9kZXYgKnBkZXYsIHZvaWQgKmFk ZHIpOworc3RydWN0IHNjYXR0ZXJsaXN0ICpwY2lfcDJwbWVtX2FsbG9jX3NnbChzdHJ1Y3QgcGNp X2RldiAqcGRldiwKKwkJCQkJIHVuc2lnbmVkIGludCAqbmVudHMsIHUzMiBsZW5ndGgpOwordm9p ZCBwY2lfcDJwbWVtX2ZyZWVfc2dsKHN0cnVjdCBwY2lfZGV2ICpwZGV2LCBzdHJ1Y3Qgc2NhdHRl cmxpc3QgKnNnbCk7Cit2b2lkIHBjaV9wMnBtZW1fcHVibGlzaChzdHJ1Y3QgcGNpX2RldiAqcGRl diwgYm9vbCBwdWJsaXNoKTsKKyNlbHNlIC8qIENPTkZJR19QQ0lfUDJQRE1BICovCitzdGF0aWMg aW5saW5lIGludCBwY2lfcDJwZG1hX2FkZF9yZXNvdXJjZShzdHJ1Y3QgcGNpX2RldiAqcGRldiwg aW50IGJhciwKKwkJc2l6ZV90IHNpemUsIHU2NCBvZmZzZXQpCit7CisJcmV0dXJuIC1FT1BOT1RT VVBQOworfQorc3RhdGljIGlubGluZSBpbnQgcGNpX3AycGRtYV9hZGRfY2xpZW50KHN0cnVjdCBs aXN0X2hlYWQgKmhlYWQsCisJCXN0cnVjdCBkZXZpY2UgKmRldikKK3sKKwlyZXR1cm4gMDsKK30K K3N0YXRpYyBpbmxpbmUgdm9pZCBwY2lfcDJwZG1hX3JlbW92ZV9jbGllbnQoc3RydWN0IGxpc3Rf aGVhZCAqaGVhZCwKKwkJc3RydWN0IGRldmljZSAqZGV2KQoreworfQorc3RhdGljIGlubGluZSB2 b2lkIHBjaV9wMnBkbWFfY2xpZW50X2xpc3RfZnJlZShzdHJ1Y3QgbGlzdF9oZWFkICpoZWFkKQor eworfQorc3RhdGljIGlubGluZSBpbnQgcGNpX3AycGRtYV9kaXN0YW5jZShzdHJ1Y3QgcGNpX2Rl diAqcHJvdmlkZXIsCisJCQkJICAgICAgc3RydWN0IGxpc3RfaGVhZCAqY2xpZW50cywKKwkJCQkg ICAgICBib29sIHZlcmJvc2UpCit7CisJcmV0dXJuIC0xOworfQorc3RhdGljIGlubGluZSBib29s IHBjaV9wMnBkbWFfYXNzaWduX3Byb3ZpZGVyKHN0cnVjdCBwY2lfZGV2ICpwcm92aWRlciwKKwkJ CQkJICAgICAgc3RydWN0IGxpc3RfaGVhZCAqY2xpZW50cykKK3sKKwlyZXR1cm4gZmFsc2U7Cit9 CitzdGF0aWMgaW5saW5lIGJvb2wgcGNpX2hhc19wMnBtZW0oc3RydWN0IHBjaV9kZXYgKnBkZXYp Cit7CisJcmV0dXJuIGZhbHNlOworfQorc3RhdGljIGlubGluZSBzdHJ1Y3QgcGNpX2RldiAqcGNp X3AycG1lbV9maW5kKHN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMpCit7CisJcmV0dXJuIE5VTEw7 Cit9CitzdGF0aWMgaW5saW5lIHZvaWQgKnBjaV9hbGxvY19wMnBtZW0oc3RydWN0IHBjaV9kZXYg KnBkZXYsIHNpemVfdCBzaXplKQoreworCXJldHVybiBOVUxMOworfQorc3RhdGljIGlubGluZSB2 b2lkIHBjaV9mcmVlX3AycG1lbShzdHJ1Y3QgcGNpX2RldiAqcGRldiwgdm9pZCAqYWRkciwKKwkJ c2l6ZV90IHNpemUpCit7Cit9CitzdGF0aWMgaW5saW5lIHBjaV9idXNfYWRkcl90IHBjaV9wMnBt ZW1fdmlydF90b19idXMoc3RydWN0IHBjaV9kZXYgKnBkZXYsCisJCQkJCQkgICAgdm9pZCAqYWRk cikKK3sKKwlyZXR1cm4gMDsKK30KK3N0YXRpYyBpbmxpbmUgc3RydWN0IHNjYXR0ZXJsaXN0ICpw Y2lfcDJwbWVtX2FsbG9jX3NnbChzdHJ1Y3QgcGNpX2RldiAqcGRldiwKKwkJdW5zaWduZWQgaW50 ICpuZW50cywgdTMyIGxlbmd0aCkKK3sKKwlyZXR1cm4gTlVMTDsKK30KK3N0YXRpYyBpbmxpbmUg dm9pZCBwY2lfcDJwbWVtX2ZyZWVfc2dsKHN0cnVjdCBwY2lfZGV2ICpwZGV2LAorCQlzdHJ1Y3Qg c2NhdHRlcmxpc3QgKnNnbCkKK3sKK30KK3N0YXRpYyBpbmxpbmUgdm9pZCBwY2lfcDJwbWVtX3B1 Ymxpc2goc3RydWN0IHBjaV9kZXYgKnBkZXYsIGJvb2wgcHVibGlzaCkKK3sKK30KKyNlbmRpZiAv KiBDT05GSUdfUENJX1AyUERNQSAqLworI2VuZGlmIC8qIF9MSU5VWF9QQ0lfUDJQX0ggKi8KZGlm ZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvcGNpLmggYi9pbmNsdWRlL2xpbnV4L3BjaS5oCmluZGV4 IGU3MmNhOGRkNjI0MS4uNWQ5NWRiZjIxZjRhIDEwMDY0NAotLS0gYS9pbmNsdWRlL2xpbnV4L3Bj aS5oCisrKyBiL2luY2x1ZGUvbGludXgvcGNpLmgKQEAgLTI4MSw2ICsyODEsNyBAQCBzdHJ1Y3Qg cGNpZV9saW5rX3N0YXRlOwogc3RydWN0IHBjaV92cGQ7CiBzdHJ1Y3QgcGNpX3NyaW92Owogc3Ry dWN0IHBjaV9hdHM7CitzdHJ1Y3QgcGNpX3AycGRtYTsKIAogLyogVGhlIHBjaV9kZXYgc3RydWN0 dXJlIGRlc2NyaWJlcyBQQ0kgZGV2aWNlcyAqLwogc3RydWN0IHBjaV9kZXYgewpAQCAtNDM4LDYg KzQzOSw5IEBAIHN0cnVjdCBwY2lfZGV2IHsKICNlbmRpZgogI2lmZGVmIENPTkZJR19QQ0lfUEFT SUQKIAl1MTYJCXBhc2lkX2ZlYXR1cmVzOworI2VuZGlmCisjaWZkZWYgQ09ORklHX1BDSV9QMlBE TUEKKwlzdHJ1Y3QgcGNpX3AycGRtYSAqcDJwZG1hOwogI2VuZGlmCiAJcGh5c19hZGRyX3QJcm9t OwkJLyogUGh5c2ljYWwgYWRkcmVzcyBpZiBub3QgZnJvbSBCQVIgKi8KIAlzaXplX3QJCXJvbWxl bjsJCS8qIExlbmd0aCBpZiBub3QgZnJvbSBCQVIgKi8KLS0gCjIuMTkuMAoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTGludXgtbnZkaW1tIG1haWxpbmcg bGlzdApMaW51eC1udmRpbW1AbGlzdHMuMDEub3JnCmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxt YW4vbGlzdGluZm8vbGludXgtbnZkaW1tCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Stephen Bates , Christoph Hellwig , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?q?Christian=20K=C3=B6nig?= , Jens Axboe , Logan Gunthorpe Date: Wed, 12 Sep 2018 18:11:44 -0600 Message-Id: <20180913001156.4115-2-logang@deltatee.com> In-Reply-To: <20180913001156.4115-1-logang@deltatee.com> References: <20180913001156.4115-1-logang@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Subject: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory List-ID: Some PCI devices may have memory mapped in a BAR space that's intended for use in peer-to-peer transactions. In order to enable such transactions the memory must be registered with ZONE_DEVICE pages so it can be used by DMA interfaces in existing drivers. Add an interface for other subsystems to find and allocate chunks of P2P memory as necessary to facilitate transfers between two PCI peers: int pci_p2pdma_add_client(); struct pci_dev *pci_p2pmem_find(); void *pci_alloc_p2pmem(); The new interface requires a driver to collect a list of client devices involved in the transaction with the pci_p2pmem_add_client*() functions then call pci_p2pmem_find() to obtain any suitable P2P memory. Once this is done the list is bound to the memory and the calling driver is free to add and remove clients as necessary (adding incompatible clients will fail). With a suitable p2pmem device, memory can then be allocated with pci_alloc_p2pmem() for use in DMA transactions. Depending on hardware, using peer-to-peer memory may reduce the bandwidth of the transfer but can significantly reduce pressure on system memory. This may be desirable in many cases: for example a system could be designed with a small CPU connected to a PCIe switch by a small number of lanes which would maximize the number of lanes available to connect to NVMe devices. The code is designed to only utilize the p2pmem device if all the devices involved in a transfer are behind the same PCI bridge. This is because we have no way of knowing whether peer-to-peer routing between PCIe Root Ports is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P transfers that go through the RC is limited to only reducing DRAM usage and, in some cases, coding convenience. The PCI-SIG may be exploring adding a new capability bit to advertise whether this is possible for future hardware. This commit includes significant rework and feedback from Christoph Hellwig. Signed-off-by: Christoph Hellwig Signed-off-by: Logan Gunthorpe --- drivers/pci/Kconfig | 17 + drivers/pci/Makefile | 1 + drivers/pci/p2pdma.c | 761 +++++++++++++++++++++++++++++++++++++ include/linux/memremap.h | 5 + include/linux/mm.h | 18 + include/linux/pci-p2pdma.h | 102 +++++ include/linux/pci.h | 4 + 7 files changed, 908 insertions(+) create mode 100644 drivers/pci/p2pdma.c create mode 100644 include/linux/pci-p2pdma.h diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 56ff8f6d31fc..deb68be4fdac 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -132,6 +132,23 @@ config PCI_PASID If unsure, say N. +config PCI_P2PDMA + bool "PCI peer-to-peer transfer support" + depends on PCI && ZONE_DEVICE + select GENERIC_ALLOCATOR + help + Enableѕ drivers to do PCI peer-to-peer transactions to and from + BARs that are exposed in other devices that are the part of + the hierarchy where peer-to-peer DMA is guaranteed by the PCI + specification to work (ie. anything below a single PCI bridge). + + Many PCIe root complexes do not support P2P transactions and + it's hard to tell which support it at all, so at this time, + P2P DMA transations must be between devices behind the same root + port. + + If unsure, say N. + config PCI_LABEL def_bool y if (DMI || ACPI) depends on PCI diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile index 1b2cfe51e8d7..85f4a703b2be 100644 --- a/drivers/pci/Makefile +++ b/drivers/pci/Makefile @@ -26,6 +26,7 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o obj-$(CONFIG_PCI_STUB) += pci-stub.o obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o obj-$(CONFIG_PCI_ECAM) += ecam.o +obj-$(CONFIG_PCI_P2PDMA) += p2pdma.o obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o # Endpoint library must be initialized before its users diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c new file mode 100644 index 000000000000..88aaec5351cd --- /dev/null +++ b/drivers/pci/p2pdma.c @@ -0,0 +1,761 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * PCI Peer 2 Peer DMA support. + * + * Copyright (c) 2016-2018, Logan Gunthorpe + * Copyright (c) 2016-2017, Microsemi Corporation + * Copyright (c) 2017, Christoph Hellwig + * Copyright (c) 2018, Eideticom Inc. + */ + +#define pr_fmt(fmt) "pci-p2pdma: " fmt +#include +#include +#include +#include +#include +#include +#include +#include + +struct pci_p2pdma { + struct percpu_ref devmap_ref; + struct completion devmap_ref_done; + struct gen_pool *pool; + bool p2pmem_published; +}; + +static void pci_p2pdma_percpu_release(struct percpu_ref *ref) +{ + struct pci_p2pdma *p2p = + container_of(ref, struct pci_p2pdma, devmap_ref); + + complete_all(&p2p->devmap_ref_done); +} + +static void pci_p2pdma_percpu_kill(void *data) +{ + struct percpu_ref *ref = data; + + if (percpu_ref_is_dying(ref)) + return; + + percpu_ref_kill(ref); +} + +static void pci_p2pdma_release(void *data) +{ + struct pci_dev *pdev = data; + + if (!pdev->p2pdma) + return; + + wait_for_completion(&pdev->p2pdma->devmap_ref_done); + percpu_ref_exit(&pdev->p2pdma->devmap_ref); + + gen_pool_destroy(pdev->p2pdma->pool); + pdev->p2pdma = NULL; +} + +static int pci_p2pdma_setup(struct pci_dev *pdev) +{ + int error = -ENOMEM; + struct pci_p2pdma *p2p; + + p2p = devm_kzalloc(&pdev->dev, sizeof(*p2p), GFP_KERNEL); + if (!p2p) + return -ENOMEM; + + p2p->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev)); + if (!p2p->pool) + goto out; + + init_completion(&p2p->devmap_ref_done); + error = percpu_ref_init(&p2p->devmap_ref, + pci_p2pdma_percpu_release, 0, GFP_KERNEL); + if (error) + goto out_pool_destroy; + + percpu_ref_switch_to_atomic_sync(&p2p->devmap_ref); + + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev); + if (error) + goto out_pool_destroy; + + pdev->p2pdma = p2p; + + return 0; + +out_pool_destroy: + gen_pool_destroy(p2p->pool); +out: + devm_kfree(&pdev->dev, p2p); + return error; +} + +/** + * pci_p2pdma_add_resource - add memory for use as p2p memory + * @pdev: the device to add the memory to + * @bar: PCI BAR to add + * @size: size of the memory to add, may be zero to use the whole BAR + * @offset: offset into the PCI BAR + * + * The memory will be given ZONE_DEVICE struct pages so that it may + * be used with any DMA request. + */ +int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, + u64 offset) +{ + struct dev_pagemap *pgmap; + void *addr; + int error; + + if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM)) + return -EINVAL; + + if (offset >= pci_resource_len(pdev, bar)) + return -EINVAL; + + if (!size) + size = pci_resource_len(pdev, bar) - offset; + + if (size + offset > pci_resource_len(pdev, bar)) + return -EINVAL; + + if (!pdev->p2pdma) { + error = pci_p2pdma_setup(pdev); + if (error) + return error; + } + + pgmap = devm_kzalloc(&pdev->dev, sizeof(*pgmap), GFP_KERNEL); + if (!pgmap) + return -ENOMEM; + + pgmap->res.start = pci_resource_start(pdev, bar) + offset; + pgmap->res.end = pgmap->res.start + size - 1; + pgmap->res.flags = pci_resource_flags(pdev, bar); + pgmap->ref = &pdev->p2pdma->devmap_ref; + pgmap->type = MEMORY_DEVICE_PCI_P2PDMA; + + addr = devm_memremap_pages(&pdev->dev, pgmap); + if (IS_ERR(addr)) { + error = PTR_ERR(addr); + goto pgmap_free; + } + + error = gen_pool_add_virt(pdev->p2pdma->pool, (unsigned long)addr, + pci_bus_address(pdev, bar) + offset, + resource_size(&pgmap->res), dev_to_node(&pdev->dev)); + if (error) + goto pgmap_free; + + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_percpu_kill, + &pdev->p2pdma->devmap_ref); + if (error) + goto pgmap_free; + + pci_info(pdev, "added peer-to-peer DMA memory %pR\n", + &pgmap->res); + + return 0; + +pgmap_free: + devres_free(pgmap); + return error; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_add_resource); + +static struct pci_dev *find_parent_pci_dev(struct device *dev) +{ + struct device *parent; + + dev = get_device(dev); + + while (dev) { + if (dev_is_pci(dev)) + return to_pci_dev(dev); + + parent = get_device(dev->parent); + put_device(dev); + dev = parent; + } + + return NULL; +} + +/* + * Check if a PCI bridge has it's ACS redirection bits set to redirect P2P + * TLPs upstream via ACS. Returns 1 if the packets will be redirected + * upstream, 0 otherwise. + */ +static int pci_bridge_has_acs_redir(struct pci_dev *dev) +{ + int pos; + u16 ctrl; + + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS); + if (!pos) + return 0; + + pci_read_config_word(dev, pos + PCI_ACS_CTRL, &ctrl); + + if (ctrl & (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC)) + return 1; + + return 0; +} + +static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *dev) +{ + if (!buf) + return; + + seq_buf_printf(buf, "%04x:%02x:%02x.%x;", pci_domain_nr(dev->bus), + dev->bus->number, PCI_SLOT(dev->devfn), + PCI_FUNC(dev->devfn)); +} + +/* + * Find the distance through the nearest common upstream bridge between + * two PCI devices. + * + * If the two devices are the same device then 0 will be returned. + * + * If there are two virtual functions of the same device behind the same + * bridge port then 2 will be returned (one step down to the PCIe switch, + * then one step back to the same device). + * + * In the case where two devices are connected to the same PCIe switch, the + * value 4 will be returned. This corresponds to the following PCI tree: + * + * -+ Root Port + * \+ Switch Upstream Port + * +-+ Switch Downstream Port + * + \- Device A + * \-+ Switch Downstream Port + * \- Device B + * + * The distance is 4 because we traverse from Device A through the downstream + * port of the switch, to the common upstream port, back up to the second + * downstream port and then to Device B. + * + * Any two devices that don't have a common upstream bridge will return -1. + * In this way devices on separate PCIe root ports will be rejected, which + * is what we want for peer-to-peer seeing each PCIe root port defines a + * separate hierarchy domain and there's no way to determine whether the root + * complex supports forwarding between them. + * + * In the case where two devices are connected to different PCIe switches, + * this function will still return a positive distance as long as both + * switches evenutally have a common upstream bridge. Note this covers + * the case of using multiple PCIe switches to achieve a desired level of + * fan-out from a root port. The exact distance will be a function of the + * number of switches between Device A and Device B. + * + * If a bridge which has any ACS redirection bits set is in the path + * then this functions will return -2. This is so we reject any + * cases where the TLPs are forwarded up into the root complex. + * In this case, a list of all infringing bridge addresses will be + * populated in acs_list (assuming it's non-null) for printk purposes. + */ +static int upstream_bridge_distance(struct pci_dev *a, + struct pci_dev *b, + struct seq_buf *acs_list) +{ + int dist_a = 0; + int dist_b = 0; + struct pci_dev *bb = NULL; + int acs_cnt = 0; + + /* + * Note, we don't need to take references to devices returned by + * pci_upstream_bridge() seeing we hold a reference to a child + * device which will already hold a reference to the upstream bridge. + */ + + while (a) { + dist_b = 0; + + if (pci_bridge_has_acs_redir(a)) { + seq_buf_print_bus_devfn(acs_list, a); + acs_cnt++; + } + + bb = b; + + while (bb) { + if (a == bb) + goto check_b_path_acs; + + bb = pci_upstream_bridge(bb); + dist_b++; + } + + a = pci_upstream_bridge(a); + dist_a++; + } + + return -1; + +check_b_path_acs: + bb = b; + + while (bb) { + if (a == bb) + break; + + if (pci_bridge_has_acs_redir(bb)) { + seq_buf_print_bus_devfn(acs_list, bb); + acs_cnt++; + } + + bb = pci_upstream_bridge(bb); + } + + if (acs_cnt) + return -2; + + return dist_a + dist_b; +} + +static int upstream_bridge_distance_warn(struct pci_dev *provider, + struct pci_dev *client) +{ + struct seq_buf acs_list; + int ret; + + seq_buf_init(&acs_list, kmalloc(PAGE_SIZE, GFP_KERNEL), PAGE_SIZE); + + ret = upstream_bridge_distance(provider, client, &acs_list); + if (ret == -2) { + pci_warn(client, "cannot be used for peer-to-peer DMA as ACS redirect is set between the client and provider\n"); + /* Drop final semicolon */ + acs_list.buffer[acs_list.len-1] = 0; + pci_warn(client, "to disable ACS redirect for this path, add the kernel parameter: pci=disable_acs_redir=%s\n", + acs_list.buffer); + + } else if (ret < 0) { + pci_warn(client, "cannot be used for peer-to-peer DMA as the client and provider do not share an upstream bridge\n"); + } + + kfree(acs_list.buffer); + + return ret; +} + +struct pci_p2pdma_client { + struct list_head list; + struct pci_dev *client; + struct pci_dev *provider; +}; + +/** + * pci_p2pdma_add_client - allocate a new element in a client device list + * @head: list head of p2pdma clients + * @dev: device to add to the list + * + * This adds @dev to a list of clients used by a p2pdma device. + * This list should be passed to pci_p2pmem_find(). Once pci_p2pmem_find() has + * been called successfully, the list will be bound to a specific p2pdma + * device and new clients can only be added to the list if they are + * supported by that p2pdma device. + * + * The caller is expected to have a lock which protects @head as necessary + * so that none of the pci_p2p functions can be called concurrently + * on that list. + * + * Returns 0 if the client was successfully added. + */ +int pci_p2pdma_add_client(struct list_head *head, struct device *dev) +{ + struct pci_p2pdma_client *item, *new_item; + struct pci_dev *provider = NULL; + struct pci_dev *client; + int ret; + + if (IS_ENABLED(CONFIG_DMA_VIRT_OPS) && dev->dma_ops == &dma_virt_ops) { + dev_warn(dev, "cannot be used for peer-to-peer DMA because the driver makes use of dma_virt_ops\n"); + return -ENODEV; + } + + client = find_parent_pci_dev(dev); + if (!client) { + dev_warn(dev, "cannot be used for peer-to-peer DMA as it is not a PCI device\n"); + return -ENODEV; + } + + item = list_first_entry_or_null(head, struct pci_p2pdma_client, list); + if (item && item->provider) { + provider = item->provider; + + ret = upstream_bridge_distance_warn(provider, client); + if (ret < 0) { + ret = -EXDEV; + goto put_client; + } + } + + new_item = kzalloc(sizeof(*new_item), GFP_KERNEL); + if (!new_item) { + ret = -ENOMEM; + goto put_client; + } + + new_item->client = client; + new_item->provider = pci_dev_get(provider); + + list_add_tail(&new_item->list, head); + + return 0; + +put_client: + pci_dev_put(client); + return ret; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_add_client); + +static void pci_p2pdma_client_free(struct pci_p2pdma_client *item) +{ + list_del(&item->list); + pci_dev_put(item->client); + pci_dev_put(item->provider); + kfree(item); +} + +/** + * pci_p2pdma_remove_client - remove and free a p2pdma client + * @head: list head of p2pdma clients + * @dev: device to remove from the list + * + * This removes @dev from a list of clients used by a p2pdma device. + * The caller is expected to have a lock which protects @head as necessary + * so that none of the pci_p2p functions can be called concurrently + * on that list. + */ +void pci_p2pdma_remove_client(struct list_head *head, struct device *dev) +{ + struct pci_p2pdma_client *pos, *tmp; + struct pci_dev *pdev; + + pdev = find_parent_pci_dev(dev); + if (!pdev) + return; + + list_for_each_entry_safe(pos, tmp, head, list) { + if (pos->client != pdev) + continue; + + pci_p2pdma_client_free(pos); + } + + pci_dev_put(pdev); +} +EXPORT_SYMBOL_GPL(pci_p2pdma_remove_client); + +/** + * pci_p2pdma_client_list_free - free an entire list of p2pdma clients + * @head: list head of p2pdma clients + * + * This removes all devices in a list of clients used by a p2pdma device. + * The caller is expected to have a lock which protects @head as necessary + * so that none of the pci_p2pdma functions can be called concurrently + * on that list. + */ +void pci_p2pdma_client_list_free(struct list_head *head) +{ + struct pci_p2pdma_client *pos, *tmp; + + list_for_each_entry_safe(pos, tmp, head, list) + pci_p2pdma_client_free(pos); +} +EXPORT_SYMBOL_GPL(pci_p2pdma_client_list_free); + +/** + * pci_p2pdma_distance - Determive the cumulative distance between + * a p2pdma provider and the clients in use. + * @provider: p2pdma provider to check against the client list + * @clients: list of devices to check (NULL-terminated) + * @verbose: if true, print warnings for devices when we return -1 + * + * Returns -1 if any of the clients are not compatible (behind the same + * root port as the provider), otherwise returns a positive number where + * the lower number is the preferrable choice. (If there's one client + * that's the same as the provider it will return 0, which is best choice). + * + * For now, "compatible" means the provider and the clients are all behind + * the same PCI root port. This cuts out cases that may work but is safest + * for the user. Future work can expand this to white-list root complexes that + * can safely forward between each ports. + */ +int pci_p2pdma_distance(struct pci_dev *provider, struct list_head *clients, + bool verbose) +{ + struct pci_p2pdma_client *pos; + int ret; + int distance = 0; + bool not_supported = false; + + if (list_empty(clients)) + return -1; + + list_for_each_entry(pos, clients, list) { + if (verbose) + ret = upstream_bridge_distance_warn(provider, + pos->client); + else + ret = upstream_bridge_distance(provider, pos->client, + NULL); + + if (ret < 0) + not_supported = true; + + if (not_supported && !verbose) + break; + + distance += ret; + } + + if (not_supported) + return -1; + + return distance; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_distance); + +/** + * pci_p2pdma_assign_provider - Check compatibily (as per pci_p2pdma_distance) + * and assign a provider to a list of clients + * @provider: p2pdma provider to assign to the client list + * @clients: list of devices to check (NULL-terminated) + * + * Returns false if any of the clients are not compatible, true if the + * provider was successfully assigned to the clients. + */ +bool pci_p2pdma_assign_provider(struct pci_dev *provider, + struct list_head *clients) +{ + struct pci_p2pdma_client *pos; + + if (pci_p2pdma_distance(provider, clients, true) < 0) + return false; + + list_for_each_entry(pos, clients, list) + pos->provider = provider; + + return true; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_assign_provider); + +/** + * pci_has_p2pmem - check if a given PCI device has published any p2pmem + * @pdev: PCI device to check + */ +bool pci_has_p2pmem(struct pci_dev *pdev) +{ + return pdev->p2pdma && pdev->p2pdma->p2pmem_published; +} +EXPORT_SYMBOL_GPL(pci_has_p2pmem); + +/** + * pci_p2pmem_find - find a peer-to-peer DMA memory device compatible with + * the specified list of clients and shortest distance (as determined + * by pci_p2pmem_dma()) + * @clients: list of devices to check (NULL-terminated) + * + * If multiple devices are behind the same switch, the one "closest" to the + * client devices in use will be chosen first. (So if one of the providers are + * the same as one of the clients, that provider will be used ahead of any + * other providers that are unrelated). If multiple providers are an equal + * distance away, one will be chosen at random. + * + * Returns a pointer to the PCI device with a reference taken (use pci_dev_put + * to return the reference) or NULL if no compatible device is found. The + * found provider will also be assigned to the client list. + */ +struct pci_dev *pci_p2pmem_find(struct list_head *clients) +{ + struct pci_dev *pdev = NULL; + struct pci_p2pdma_client *pos; + int distance; + int closest_distance = INT_MAX; + struct pci_dev **closest_pdevs; + int dev_cnt = 0; + const int max_devs = PAGE_SIZE / sizeof(*closest_pdevs); + int i; + + closest_pdevs = kmalloc(PAGE_SIZE, GFP_KERNEL); + + while ((pdev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, pdev))) { + if (!pci_has_p2pmem(pdev)) + continue; + + distance = pci_p2pdma_distance(pdev, clients, false); + if (distance < 0 || distance > closest_distance) + continue; + + if (distance == closest_distance && dev_cnt >= max_devs) + continue; + + if (distance < closest_distance) { + for (i = 0; i < dev_cnt; i++) + pci_dev_put(closest_pdevs[i]); + + dev_cnt = 0; + closest_distance = distance; + } + + closest_pdevs[dev_cnt++] = pci_dev_get(pdev); + } + + if (dev_cnt) + pdev = pci_dev_get(closest_pdevs[prandom_u32_max(dev_cnt)]); + + for (i = 0; i < dev_cnt; i++) + pci_dev_put(closest_pdevs[i]); + + if (pdev) + list_for_each_entry(pos, clients, list) + pos->provider = pdev; + + kfree(closest_pdevs); + return pdev; +} +EXPORT_SYMBOL_GPL(pci_p2pmem_find); + +/** + * pci_alloc_p2p_mem - allocate peer-to-peer DMA memory + * @pdev: the device to allocate memory from + * @size: number of bytes to allocate + * + * Returns the allocated memory or NULL on error. + */ +void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size) +{ + void *ret; + + if (unlikely(!pdev->p2pdma)) + return NULL; + + if (unlikely(!percpu_ref_tryget_live(&pdev->p2pdma->devmap_ref))) + return NULL; + + ret = (void *)gen_pool_alloc(pdev->p2pdma->pool, size); + + if (unlikely(!ret)) + percpu_ref_put(&pdev->p2pdma->devmap_ref); + + return ret; +} +EXPORT_SYMBOL_GPL(pci_alloc_p2pmem); + +/** + * pci_free_p2pmem - allocate peer-to-peer DMA memory + * @pdev: the device the memory was allocated from + * @addr: address of the memory that was allocated + * @size: number of bytes that was allocated + */ +void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size) +{ + gen_pool_free(pdev->p2pdma->pool, (uintptr_t)addr, size); + percpu_ref_put(&pdev->p2pdma->devmap_ref); +} +EXPORT_SYMBOL_GPL(pci_free_p2pmem); + +/** + * pci_virt_to_bus - return the PCI bus address for a given virtual + * address obtained with pci_alloc_p2pmem() + * @pdev: the device the memory was allocated from + * @addr: address of the memory that was allocated + */ +pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr) +{ + if (!addr) + return 0; + if (!pdev->p2pdma) + return 0; + + /* + * Note: when we added the memory to the pool we used the PCI + * bus address as the physical address. So gen_pool_virt_to_phys() + * actually returns the bus address despite the misleading name. + */ + return gen_pool_virt_to_phys(pdev->p2pdma->pool, (unsigned long)addr); +} +EXPORT_SYMBOL_GPL(pci_p2pmem_virt_to_bus); + +/** + * pci_p2pmem_alloc_sgl - allocate peer-to-peer DMA memory in a scatterlist + * @pdev: the device to allocate memory from + * @sgl: the allocated scatterlist + * @nents: the number of SG entries in the list + * @length: number of bytes to allocate + * + * Returns 0 on success + */ +struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, + unsigned int *nents, u32 length) +{ + struct scatterlist *sg; + void *addr; + + sg = kzalloc(sizeof(*sg), GFP_KERNEL); + if (!sg) + return NULL; + + sg_init_table(sg, 1); + + addr = pci_alloc_p2pmem(pdev, length); + if (!addr) + goto out_free_sg; + + sg_set_buf(sg, addr, length); + *nents = 1; + return sg; + +out_free_sg: + kfree(sg); + return NULL; +} +EXPORT_SYMBOL_GPL(pci_p2pmem_alloc_sgl); + +/** + * pci_p2pmem_free_sgl - free a scatterlist allocated by pci_p2pmem_alloc_sgl() + * @pdev: the device to allocate memory from + * @sgl: the allocated scatterlist + * @nents: the number of SG entries in the list + */ +void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl) +{ + struct scatterlist *sg; + int count; + + for_each_sg(sgl, sg, INT_MAX, count) { + if (!sg) + break; + + pci_free_p2pmem(pdev, sg_virt(sg), sg->length); + } + kfree(sgl); +} +EXPORT_SYMBOL_GPL(pci_p2pmem_free_sgl); + +/** + * pci_p2pmem_publish - publish the peer-to-peer DMA memory for use by + * other devices with pci_p2pmem_find() + * @pdev: the device with peer-to-peer DMA memory to publish + * @publish: set to true to publish the memory, false to unpublish it + * + * Published memory can be used by other PCI device drivers for + * peer-2-peer DMA operations. Non-published memory is reserved for + * exlusive use of the device driver that registers the peer-to-peer + * memory. + */ +void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) +{ + if (publish && !pdev->p2pdma) + return; + + pdev->p2pdma->p2pmem_published = publish; +} +EXPORT_SYMBOL_GPL(pci_p2pmem_publish); diff --git a/include/linux/memremap.h b/include/linux/memremap.h index f91f9e763557..9553370ebdad 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -53,11 +53,16 @@ struct vmem_altmap { * wakeup event whenever a page is unpinned and becomes idle. This * wakeup is used to coordinate physical address space management (ex: * fs truncate/hole punch) vs pinned pages (ex: device dma). + * + * MEMORY_DEVICE_PCI_P2PDMA: + * Device memory residing in a PCI BAR intended for use with Peer-to-Peer + * transactions. */ enum memory_type { MEMORY_DEVICE_PRIVATE = 1, MEMORY_DEVICE_PUBLIC, MEMORY_DEVICE_FS_DAX, + MEMORY_DEVICE_PCI_P2PDMA, }; /* diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..2055df412a77 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -890,6 +890,19 @@ static inline bool is_device_public_page(const struct page *page) page->pgmap->type == MEMORY_DEVICE_PUBLIC; } +#ifdef CONFIG_PCI_P2PDMA +static inline bool is_pci_p2pdma_page(const struct page *page) +{ + return is_zone_device_page(page) && + page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA; +} +#else /* CONFIG_PCI_P2PDMA */ +static inline bool is_pci_p2pdma_page(const struct page *page) +{ + return false; +} +#endif /* CONFIG_PCI_P2PDMA */ + #else /* CONFIG_DEV_PAGEMAP_OPS */ static inline void dev_pagemap_get_ops(void) { @@ -913,6 +926,11 @@ static inline bool is_device_public_page(const struct page *page) { return false; } + +static inline bool is_pci_p2pdma_page(const struct page *page) +{ + return false; +} #endif /* CONFIG_DEV_PAGEMAP_OPS */ static inline void get_page(struct page *page) diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h new file mode 100644 index 000000000000..7b2b0f547528 --- /dev/null +++ b/include/linux/pci-p2pdma.h @@ -0,0 +1,102 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * PCI Peer 2 Peer DMA support. + * + * Copyright (c) 2016-2018, Logan Gunthorpe + * Copyright (c) 2016-2017, Microsemi Corporation + * Copyright (c) 2017, Christoph Hellwig + * Copyright (c) 2018, Eideticom Inc. + * + */ + +#ifndef _LINUX_PCI_P2PDMA_H +#define _LINUX_PCI_P2PDMA_H + +#include + +struct block_device; +struct scatterlist; + +#ifdef CONFIG_PCI_P2PDMA +int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, + u64 offset); +int pci_p2pdma_add_client(struct list_head *head, struct device *dev); +void pci_p2pdma_remove_client(struct list_head *head, struct device *dev); +void pci_p2pdma_client_list_free(struct list_head *head); +int pci_p2pdma_distance(struct pci_dev *provider, struct list_head *clients, + bool verbose); +bool pci_p2pdma_assign_provider(struct pci_dev *provider, + struct list_head *clients); +bool pci_has_p2pmem(struct pci_dev *pdev); +struct pci_dev *pci_p2pmem_find(struct list_head *clients); +void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size); +void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size); +pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr); +struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, + unsigned int *nents, u32 length); +void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl); +void pci_p2pmem_publish(struct pci_dev *pdev, bool publish); +#else /* CONFIG_PCI_P2PDMA */ +static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, + size_t size, u64 offset) +{ + return -EOPNOTSUPP; +} +static inline int pci_p2pdma_add_client(struct list_head *head, + struct device *dev) +{ + return 0; +} +static inline void pci_p2pdma_remove_client(struct list_head *head, + struct device *dev) +{ +} +static inline void pci_p2pdma_client_list_free(struct list_head *head) +{ +} +static inline int pci_p2pdma_distance(struct pci_dev *provider, + struct list_head *clients, + bool verbose) +{ + return -1; +} +static inline bool pci_p2pdma_assign_provider(struct pci_dev *provider, + struct list_head *clients) +{ + return false; +} +static inline bool pci_has_p2pmem(struct pci_dev *pdev) +{ + return false; +} +static inline struct pci_dev *pci_p2pmem_find(struct list_head *clients) +{ + return NULL; +} +static inline void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size) +{ + return NULL; +} +static inline void pci_free_p2pmem(struct pci_dev *pdev, void *addr, + size_t size) +{ +} +static inline pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, + void *addr) +{ + return 0; +} +static inline struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, + unsigned int *nents, u32 length) +{ + return NULL; +} +static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev, + struct scatterlist *sgl) +{ +} +static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) +{ +} +#endif /* CONFIG_PCI_P2PDMA */ +#endif /* _LINUX_PCI_P2P_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index e72ca8dd6241..5d95dbf21f4a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -281,6 +281,7 @@ struct pcie_link_state; struct pci_vpd; struct pci_sriov; struct pci_ats; +struct pci_p2pdma; /* The pci_dev structure describes PCI devices */ struct pci_dev { @@ -438,6 +439,9 @@ struct pci_dev { #endif #ifdef CONFIG_PCI_PASID u16 pasid_features; +#endif +#ifdef CONFIG_PCI_P2PDMA + struct pci_p2pdma *p2pdma; #endif phys_addr_t rom; /* Physical address if not from BAR */ size_t romlen; /* Length if not from BAR */ -- 2.19.0 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory Date: Wed, 12 Sep 2018 18:11:44 -0600 Message-ID: <20180913001156.4115-2-logang@deltatee.com> References: <20180913001156.4115-1-logang@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20180913001156.4115-1-logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Cc: Jens Axboe , =?UTF-8?q?Christian=20K=C3=B6nig?= , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org U29tZSBQQ0kgZGV2aWNlcyBtYXkgaGF2ZSBtZW1vcnkgbWFwcGVkIGluIGEgQkFSIHNwYWNlIHRo YXQncwppbnRlbmRlZCBmb3IgdXNlIGluIHBlZXItdG8tcGVlciB0cmFuc2FjdGlvbnMuIEluIG9y ZGVyIHRvIGVuYWJsZQpzdWNoIHRyYW5zYWN0aW9ucyB0aGUgbWVtb3J5IG11c3QgYmUgcmVnaXN0 ZXJlZCB3aXRoIFpPTkVfREVWSUNFIHBhZ2VzCnNvIGl0IGNhbiBiZSB1c2VkIGJ5IERNQSBpbnRl cmZhY2VzIGluIGV4aXN0aW5nIGRyaXZlcnMuCgpBZGQgYW4gaW50ZXJmYWNlIGZvciBvdGhlciBz dWJzeXN0ZW1zIHRvIGZpbmQgYW5kIGFsbG9jYXRlIGNodW5rcyBvZiBQMlAKbWVtb3J5IGFzIG5l Y2Vzc2FyeSB0byBmYWNpbGl0YXRlIHRyYW5zZmVycyBiZXR3ZWVuIHR3byBQQ0kgcGVlcnM6Cgpp bnQgcGNpX3AycGRtYV9hZGRfY2xpZW50KCk7CnN0cnVjdCBwY2lfZGV2ICpwY2lfcDJwbWVtX2Zp bmQoKTsKdm9pZCAqcGNpX2FsbG9jX3AycG1lbSgpOwoKVGhlIG5ldyBpbnRlcmZhY2UgcmVxdWly ZXMgYSBkcml2ZXIgdG8gY29sbGVjdCBhIGxpc3Qgb2YgY2xpZW50IGRldmljZXMKaW52b2x2ZWQg aW4gdGhlIHRyYW5zYWN0aW9uIHdpdGggdGhlIHBjaV9wMnBtZW1fYWRkX2NsaWVudCooKSBmdW5j dGlvbnMKdGhlbiBjYWxsIHBjaV9wMnBtZW1fZmluZCgpIHRvIG9idGFpbiBhbnkgc3VpdGFibGUg UDJQIG1lbW9yeS4gT25jZQp0aGlzIGlzIGRvbmUgdGhlIGxpc3QgaXMgYm91bmQgdG8gdGhlIG1l bW9yeSBhbmQgdGhlIGNhbGxpbmcgZHJpdmVyIGlzCmZyZWUgdG8gYWRkIGFuZCByZW1vdmUgY2xp ZW50cyBhcyBuZWNlc3NhcnkgKGFkZGluZyBpbmNvbXBhdGlibGUgY2xpZW50cwp3aWxsIGZhaWwp LiBXaXRoIGEgc3VpdGFibGUgcDJwbWVtIGRldmljZSwgbWVtb3J5IGNhbiB0aGVuIGJlCmFsbG9j YXRlZCB3aXRoIHBjaV9hbGxvY19wMnBtZW0oKSBmb3IgdXNlIGluIERNQSB0cmFuc2FjdGlvbnMu CgpEZXBlbmRpbmcgb24gaGFyZHdhcmUsIHVzaW5nIHBlZXItdG8tcGVlciBtZW1vcnkgbWF5IHJl ZHVjZSB0aGUgYmFuZHdpZHRoCm9mIHRoZSB0cmFuc2ZlciBidXQgY2FuIHNpZ25pZmljYW50bHkg cmVkdWNlIHByZXNzdXJlIG9uIHN5c3RlbSBtZW1vcnkuClRoaXMgbWF5IGJlIGRlc2lyYWJsZSBp biBtYW55IGNhc2VzOiBmb3IgZXhhbXBsZSBhIHN5c3RlbSBjb3VsZCBiZSBkZXNpZ25lZAp3aXRo IGEgc21hbGwgQ1BVIGNvbm5lY3RlZCB0byBhIFBDSWUgc3dpdGNoIGJ5IGEgc21hbGwgbnVtYmVy IG9mIGxhbmVzCndoaWNoIHdvdWxkIG1heGltaXplIHRoZSBudW1iZXIgb2YgbGFuZXMgYXZhaWxh YmxlIHRvIGNvbm5lY3QgdG8gTlZNZQpkZXZpY2VzLgoKVGhlIGNvZGUgaXMgZGVzaWduZWQgdG8g b25seSB1dGlsaXplIHRoZSBwMnBtZW0gZGV2aWNlIGlmIGFsbCB0aGUgZGV2aWNlcwppbnZvbHZl ZCBpbiBhIHRyYW5zZmVyIGFyZSBiZWhpbmQgdGhlIHNhbWUgUENJIGJyaWRnZS4gVGhpcyBpcyBi ZWNhdXNlIHdlCmhhdmUgbm8gd2F5IG9mIGtub3dpbmcgd2hldGhlciBwZWVyLXRvLXBlZXIgcm91 dGluZyBiZXR3ZWVuIFBDSWUgUm9vdCBQb3J0cwppcyBzdXBwb3J0ZWQgKFBDSWUgcjQuMCwgc2Vj IDEuMy4xKS4gQWRkaXRpb25hbGx5LCB0aGUgYmVuZWZpdHMgb2YgUDJQCnRyYW5zZmVycyB0aGF0 IGdvIHRocm91Z2ggdGhlIFJDIGlzIGxpbWl0ZWQgdG8gb25seSByZWR1Y2luZyBEUkFNIHVzYWdl CmFuZCwgaW4gc29tZSBjYXNlcywgY29kaW5nIGNvbnZlbmllbmNlLiBUaGUgUENJLVNJRyBtYXkg YmUgZXhwbG9yaW5nCmFkZGluZyBhIG5ldyBjYXBhYmlsaXR5IGJpdCB0byBhZHZlcnRpc2Ugd2hl dGhlciB0aGlzIGlzIHBvc3NpYmxlIGZvcgpmdXR1cmUgaGFyZHdhcmUuCgpUaGlzIGNvbW1pdCBp bmNsdWRlcyBzaWduaWZpY2FudCByZXdvcmsgYW5kIGZlZWRiYWNrIGZyb20gQ2hyaXN0b3BoCkhl bGx3aWcuCgpTaWduZWQtb2ZmLWJ5OiBDaHJpc3RvcGggSGVsbHdpZyA8aGNoQGxzdC5kZT4KU2ln bmVkLW9mZi1ieTogTG9nYW4gR3VudGhvcnBlIDxsb2dhbmdAZGVsdGF0ZWUuY29tPgotLS0KIGRy aXZlcnMvcGNpL0tjb25maWcgICAgICAgIHwgIDE3ICsKIGRyaXZlcnMvcGNpL01ha2VmaWxlICAg ICAgIHwgICAxICsKIGRyaXZlcnMvcGNpL3AycGRtYS5jICAgICAgIHwgNzYxICsrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysKIGluY2x1ZGUvbGludXgvbWVtcmVtYXAuaCAgIHwg ICA1ICsKIGluY2x1ZGUvbGludXgvbW0uaCAgICAgICAgIHwgIDE4ICsKIGluY2x1ZGUvbGludXgv cGNpLXAycGRtYS5oIHwgMTAyICsrKysrCiBpbmNsdWRlL2xpbnV4L3BjaS5oICAgICAgICB8ICAg NCArCiA3IGZpbGVzIGNoYW5nZWQsIDkwOCBpbnNlcnRpb25zKCspCiBjcmVhdGUgbW9kZSAxMDA2 NDQgZHJpdmVycy9wY2kvcDJwZG1hLmMKIGNyZWF0ZSBtb2RlIDEwMDY0NCBpbmNsdWRlL2xpbnV4 L3BjaS1wMnBkbWEuaAoKZGlmZiAtLWdpdCBhL2RyaXZlcnMvcGNpL0tjb25maWcgYi9kcml2ZXJz L3BjaS9LY29uZmlnCmluZGV4IDU2ZmY4ZjZkMzFmYy4uZGViNjhiZTRmZGFjIDEwMDY0NAotLS0g YS9kcml2ZXJzL3BjaS9LY29uZmlnCisrKyBiL2RyaXZlcnMvcGNpL0tjb25maWcKQEAgLTEzMiw2 ICsxMzIsMjMgQEAgY29uZmlnIFBDSV9QQVNJRAogCiAJICBJZiB1bnN1cmUsIHNheSBOLgogCitj b25maWcgUENJX1AyUERNQQorCWJvb2wgIlBDSSBwZWVyLXRvLXBlZXIgdHJhbnNmZXIgc3VwcG9y dCIKKwlkZXBlbmRzIG9uIFBDSSAmJiBaT05FX0RFVklDRQorCXNlbGVjdCBHRU5FUklDX0FMTE9D QVRPUgorCWhlbHAKKwkgIEVuYWJsZdGVIGRyaXZlcnMgdG8gZG8gUENJIHBlZXItdG8tcGVlciB0 cmFuc2FjdGlvbnMgdG8gYW5kIGZyb20KKwkgIEJBUnMgdGhhdCBhcmUgZXhwb3NlZCBpbiBvdGhl ciBkZXZpY2VzIHRoYXQgYXJlIHRoZSBwYXJ0IG9mCisJICB0aGUgaGllcmFyY2h5IHdoZXJlIHBl ZXItdG8tcGVlciBETUEgaXMgZ3VhcmFudGVlZCBieSB0aGUgUENJCisJICBzcGVjaWZpY2F0aW9u IHRvIHdvcmsgKGllLiBhbnl0aGluZyBiZWxvdyBhIHNpbmdsZSBQQ0kgYnJpZGdlKS4KKworCSAg TWFueSBQQ0llIHJvb3QgY29tcGxleGVzIGRvIG5vdCBzdXBwb3J0IFAyUCB0cmFuc2FjdGlvbnMg YW5kCisJICBpdCdzIGhhcmQgdG8gdGVsbCB3aGljaCBzdXBwb3J0IGl0IGF0IGFsbCwgc28gYXQg dGhpcyB0aW1lLAorCSAgUDJQIERNQSB0cmFuc2F0aW9ucyBtdXN0IGJlIGJldHdlZW4gZGV2aWNl cyBiZWhpbmQgdGhlIHNhbWUgcm9vdAorCSAgcG9ydC4KKworCSAgSWYgdW5zdXJlLCBzYXkgTi4K KwogY29uZmlnIFBDSV9MQUJFTAogCWRlZl9ib29sIHkgaWYgKERNSSB8fCBBQ1BJKQogCWRlcGVu ZHMgb24gUENJCmRpZmYgLS1naXQgYS9kcml2ZXJzL3BjaS9NYWtlZmlsZSBiL2RyaXZlcnMvcGNp L01ha2VmaWxlCmluZGV4IDFiMmNmZTUxZThkNy4uODVmNGE3MDNiMmJlIDEwMDY0NAotLS0gYS9k cml2ZXJzL3BjaS9NYWtlZmlsZQorKysgYi9kcml2ZXJzL3BjaS9NYWtlZmlsZQpAQCAtMjYsNiAr MjYsNyBAQCBvYmotJChDT05GSUdfUENJX1NZU0NBTEwpCSs9IHN5c2NhbGwubwogb2JqLSQoQ09O RklHX1BDSV9TVFVCKQkJKz0gcGNpLXN0dWIubwogb2JqLSQoQ09ORklHX1BDSV9QRl9TVFVCKQkr PSBwY2ktcGYtc3R1Yi5vCiBvYmotJChDT05GSUdfUENJX0VDQU0pCQkrPSBlY2FtLm8KK29iai0k KENPTkZJR19QQ0lfUDJQRE1BKQkrPSBwMnBkbWEubwogb2JqLSQoQ09ORklHX1hFTl9QQ0lERVZf RlJPTlRFTkQpICs9IHhlbi1wY2lmcm9udC5vCiAKICMgRW5kcG9pbnQgbGlicmFyeSBtdXN0IGJl IGluaXRpYWxpemVkIGJlZm9yZSBpdHMgdXNlcnMKZGlmZiAtLWdpdCBhL2RyaXZlcnMvcGNpL3Ay cGRtYS5jIGIvZHJpdmVycy9wY2kvcDJwZG1hLmMKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXgg MDAwMDAwMDAwMDAwLi44OGFhZWM1MzUxY2QKLS0tIC9kZXYvbnVsbAorKysgYi9kcml2ZXJzL3Bj aS9wMnBkbWEuYwpAQCAtMCwwICsxLDc2MSBAQAorLy8gU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6 IEdQTC0yLjAKKy8qCisgKiBQQ0kgUGVlciAyIFBlZXIgRE1BIHN1cHBvcnQuCisgKgorICogQ29w eXJpZ2h0IChjKSAyMDE2LTIwMTgsIExvZ2FuIEd1bnRob3JwZQorICogQ29weXJpZ2h0IChjKSAy MDE2LTIwMTcsIE1pY3Jvc2VtaSBDb3Jwb3JhdGlvbgorICogQ29weXJpZ2h0IChjKSAyMDE3LCBD aHJpc3RvcGggSGVsbHdpZworICogQ29weXJpZ2h0IChjKSAyMDE4LCBFaWRldGljb20gSW5jLgor ICovCisKKyNkZWZpbmUgcHJfZm10KGZtdCkgInBjaS1wMnBkbWE6ICIgZm10CisjaW5jbHVkZSA8 bGludXgvcGNpLXAycGRtYS5oPgorI2luY2x1ZGUgPGxpbnV4L21vZHVsZS5oPgorI2luY2x1ZGUg PGxpbnV4L3NsYWIuaD4KKyNpbmNsdWRlIDxsaW51eC9nZW5hbGxvYy5oPgorI2luY2x1ZGUgPGxp bnV4L21lbXJlbWFwLmg+CisjaW5jbHVkZSA8bGludXgvcGVyY3B1LXJlZmNvdW50Lmg+CisjaW5j bHVkZSA8bGludXgvcmFuZG9tLmg+CisjaW5jbHVkZSA8bGludXgvc2VxX2J1Zi5oPgorCitzdHJ1 Y3QgcGNpX3AycGRtYSB7CisJc3RydWN0IHBlcmNwdV9yZWYgZGV2bWFwX3JlZjsKKwlzdHJ1Y3Qg Y29tcGxldGlvbiBkZXZtYXBfcmVmX2RvbmU7CisJc3RydWN0IGdlbl9wb29sICpwb29sOworCWJv b2wgcDJwbWVtX3B1Ymxpc2hlZDsKK307CisKK3N0YXRpYyB2b2lkIHBjaV9wMnBkbWFfcGVyY3B1 X3JlbGVhc2Uoc3RydWN0IHBlcmNwdV9yZWYgKnJlZikKK3sKKwlzdHJ1Y3QgcGNpX3AycGRtYSAq cDJwID0KKwkJY29udGFpbmVyX29mKHJlZiwgc3RydWN0IHBjaV9wMnBkbWEsIGRldm1hcF9yZWYp OworCisJY29tcGxldGVfYWxsKCZwMnAtPmRldm1hcF9yZWZfZG9uZSk7Cit9CisKK3N0YXRpYyB2 b2lkIHBjaV9wMnBkbWFfcGVyY3B1X2tpbGwodm9pZCAqZGF0YSkKK3sKKwlzdHJ1Y3QgcGVyY3B1 X3JlZiAqcmVmID0gZGF0YTsKKworCWlmIChwZXJjcHVfcmVmX2lzX2R5aW5nKHJlZikpCisJCXJl dHVybjsKKworCXBlcmNwdV9yZWZfa2lsbChyZWYpOworfQorCitzdGF0aWMgdm9pZCBwY2lfcDJw ZG1hX3JlbGVhc2Uodm9pZCAqZGF0YSkKK3sKKwlzdHJ1Y3QgcGNpX2RldiAqcGRldiA9IGRhdGE7 CisKKwlpZiAoIXBkZXYtPnAycGRtYSkKKwkJcmV0dXJuOworCisJd2FpdF9mb3JfY29tcGxldGlv bigmcGRldi0+cDJwZG1hLT5kZXZtYXBfcmVmX2RvbmUpOworCXBlcmNwdV9yZWZfZXhpdCgmcGRl di0+cDJwZG1hLT5kZXZtYXBfcmVmKTsKKworCWdlbl9wb29sX2Rlc3Ryb3kocGRldi0+cDJwZG1h LT5wb29sKTsKKwlwZGV2LT5wMnBkbWEgPSBOVUxMOworfQorCitzdGF0aWMgaW50IHBjaV9wMnBk bWFfc2V0dXAoc3RydWN0IHBjaV9kZXYgKnBkZXYpCit7CisJaW50IGVycm9yID0gLUVOT01FTTsK KwlzdHJ1Y3QgcGNpX3AycGRtYSAqcDJwOworCisJcDJwID0gZGV2bV9remFsbG9jKCZwZGV2LT5k ZXYsIHNpemVvZigqcDJwKSwgR0ZQX0tFUk5FTCk7CisJaWYgKCFwMnApCisJCXJldHVybiAtRU5P TUVNOworCisJcDJwLT5wb29sID0gZ2VuX3Bvb2xfY3JlYXRlKFBBR0VfU0hJRlQsIGRldl90b19u b2RlKCZwZGV2LT5kZXYpKTsKKwlpZiAoIXAycC0+cG9vbCkKKwkJZ290byBvdXQ7CisKKwlpbml0 X2NvbXBsZXRpb24oJnAycC0+ZGV2bWFwX3JlZl9kb25lKTsKKwllcnJvciA9IHBlcmNwdV9yZWZf aW5pdCgmcDJwLT5kZXZtYXBfcmVmLAorCQkJcGNpX3AycGRtYV9wZXJjcHVfcmVsZWFzZSwgMCwg R0ZQX0tFUk5FTCk7CisJaWYgKGVycm9yKQorCQlnb3RvIG91dF9wb29sX2Rlc3Ryb3k7CisKKwlw ZXJjcHVfcmVmX3N3aXRjaF90b19hdG9taWNfc3luYygmcDJwLT5kZXZtYXBfcmVmKTsKKworCWVy cm9yID0gZGV2bV9hZGRfYWN0aW9uX29yX3Jlc2V0KCZwZGV2LT5kZXYsIHBjaV9wMnBkbWFfcmVs ZWFzZSwgcGRldik7CisJaWYgKGVycm9yKQorCQlnb3RvIG91dF9wb29sX2Rlc3Ryb3k7CisKKwlw ZGV2LT5wMnBkbWEgPSBwMnA7CisKKwlyZXR1cm4gMDsKKworb3V0X3Bvb2xfZGVzdHJveToKKwln ZW5fcG9vbF9kZXN0cm95KHAycC0+cG9vbCk7CitvdXQ6CisJZGV2bV9rZnJlZSgmcGRldi0+ZGV2 LCBwMnApOworCXJldHVybiBlcnJvcjsKK30KKworLyoqCisgKiBwY2lfcDJwZG1hX2FkZF9yZXNv dXJjZSAtIGFkZCBtZW1vcnkgZm9yIHVzZSBhcyBwMnAgbWVtb3J5CisgKiBAcGRldjogdGhlIGRl dmljZSB0byBhZGQgdGhlIG1lbW9yeSB0bworICogQGJhcjogUENJIEJBUiB0byBhZGQKKyAqIEBz aXplOiBzaXplIG9mIHRoZSBtZW1vcnkgdG8gYWRkLCBtYXkgYmUgemVybyB0byB1c2UgdGhlIHdo b2xlIEJBUgorICogQG9mZnNldDogb2Zmc2V0IGludG8gdGhlIFBDSSBCQVIKKyAqCisgKiBUaGUg bWVtb3J5IHdpbGwgYmUgZ2l2ZW4gWk9ORV9ERVZJQ0Ugc3RydWN0IHBhZ2VzIHNvIHRoYXQgaXQg bWF5CisgKiBiZSB1c2VkIHdpdGggYW55IERNQSByZXF1ZXN0LgorICovCitpbnQgcGNpX3AycGRt YV9hZGRfcmVzb3VyY2Uoc3RydWN0IHBjaV9kZXYgKnBkZXYsIGludCBiYXIsIHNpemVfdCBzaXpl LAorCQkJICAgIHU2NCBvZmZzZXQpCit7CisJc3RydWN0IGRldl9wYWdlbWFwICpwZ21hcDsKKwl2 b2lkICphZGRyOworCWludCBlcnJvcjsKKworCWlmICghKHBjaV9yZXNvdXJjZV9mbGFncyhwZGV2 LCBiYXIpICYgSU9SRVNPVVJDRV9NRU0pKQorCQlyZXR1cm4gLUVJTlZBTDsKKworCWlmIChvZmZz ZXQgPj0gcGNpX3Jlc291cmNlX2xlbihwZGV2LCBiYXIpKQorCQlyZXR1cm4gLUVJTlZBTDsKKwor CWlmICghc2l6ZSkKKwkJc2l6ZSA9IHBjaV9yZXNvdXJjZV9sZW4ocGRldiwgYmFyKSAtIG9mZnNl dDsKKworCWlmIChzaXplICsgb2Zmc2V0ID4gcGNpX3Jlc291cmNlX2xlbihwZGV2LCBiYXIpKQor CQlyZXR1cm4gLUVJTlZBTDsKKworCWlmICghcGRldi0+cDJwZG1hKSB7CisJCWVycm9yID0gcGNp X3AycGRtYV9zZXR1cChwZGV2KTsKKwkJaWYgKGVycm9yKQorCQkJcmV0dXJuIGVycm9yOworCX0K KworCXBnbWFwID0gZGV2bV9remFsbG9jKCZwZGV2LT5kZXYsIHNpemVvZigqcGdtYXApLCBHRlBf S0VSTkVMKTsKKwlpZiAoIXBnbWFwKQorCQlyZXR1cm4gLUVOT01FTTsKKworCXBnbWFwLT5yZXMu c3RhcnQgPSBwY2lfcmVzb3VyY2Vfc3RhcnQocGRldiwgYmFyKSArIG9mZnNldDsKKwlwZ21hcC0+ cmVzLmVuZCA9IHBnbWFwLT5yZXMuc3RhcnQgKyBzaXplIC0gMTsKKwlwZ21hcC0+cmVzLmZsYWdz ID0gcGNpX3Jlc291cmNlX2ZsYWdzKHBkZXYsIGJhcik7CisJcGdtYXAtPnJlZiA9ICZwZGV2LT5w MnBkbWEtPmRldm1hcF9yZWY7CisJcGdtYXAtPnR5cGUgPSBNRU1PUllfREVWSUNFX1BDSV9QMlBE TUE7CisKKwlhZGRyID0gZGV2bV9tZW1yZW1hcF9wYWdlcygmcGRldi0+ZGV2LCBwZ21hcCk7CisJ aWYgKElTX0VSUihhZGRyKSkgeworCQllcnJvciA9IFBUUl9FUlIoYWRkcik7CisJCWdvdG8gcGdt YXBfZnJlZTsKKwl9CisKKwllcnJvciA9IGdlbl9wb29sX2FkZF92aXJ0KHBkZXYtPnAycGRtYS0+ cG9vbCwgKHVuc2lnbmVkIGxvbmcpYWRkciwKKwkJCXBjaV9idXNfYWRkcmVzcyhwZGV2LCBiYXIp ICsgb2Zmc2V0LAorCQkJcmVzb3VyY2Vfc2l6ZSgmcGdtYXAtPnJlcyksIGRldl90b19ub2RlKCZw ZGV2LT5kZXYpKTsKKwlpZiAoZXJyb3IpCisJCWdvdG8gcGdtYXBfZnJlZTsKKworCWVycm9yID0g ZGV2bV9hZGRfYWN0aW9uX29yX3Jlc2V0KCZwZGV2LT5kZXYsIHBjaV9wMnBkbWFfcGVyY3B1X2tp bGwsCisJCQkJCSAgJnBkZXYtPnAycGRtYS0+ZGV2bWFwX3JlZik7CisJaWYgKGVycm9yKQorCQln b3RvIHBnbWFwX2ZyZWU7CisKKwlwY2lfaW5mbyhwZGV2LCAiYWRkZWQgcGVlci10by1wZWVyIERN QSBtZW1vcnkgJXBSXG4iLAorCQkgJnBnbWFwLT5yZXMpOworCisJcmV0dXJuIDA7CisKK3BnbWFw X2ZyZWU6CisJZGV2cmVzX2ZyZWUocGdtYXApOworCXJldHVybiBlcnJvcjsKK30KK0VYUE9SVF9T WU1CT0xfR1BMKHBjaV9wMnBkbWFfYWRkX3Jlc291cmNlKTsKKworc3RhdGljIHN0cnVjdCBwY2lf ZGV2ICpmaW5kX3BhcmVudF9wY2lfZGV2KHN0cnVjdCBkZXZpY2UgKmRldikKK3sKKwlzdHJ1Y3Qg ZGV2aWNlICpwYXJlbnQ7CisKKwlkZXYgPSBnZXRfZGV2aWNlKGRldik7CisKKwl3aGlsZSAoZGV2 KSB7CisJCWlmIChkZXZfaXNfcGNpKGRldikpCisJCQlyZXR1cm4gdG9fcGNpX2RldihkZXYpOwor CisJCXBhcmVudCA9IGdldF9kZXZpY2UoZGV2LT5wYXJlbnQpOworCQlwdXRfZGV2aWNlKGRldik7 CisJCWRldiA9IHBhcmVudDsKKwl9CisKKwlyZXR1cm4gTlVMTDsKK30KKworLyoKKyAqIENoZWNr IGlmIGEgUENJIGJyaWRnZSBoYXMgaXQncyBBQ1MgcmVkaXJlY3Rpb24gYml0cyBzZXQgdG8gcmVk aXJlY3QgUDJQCisgKiBUTFBzIHVwc3RyZWFtIHZpYSBBQ1MuIFJldHVybnMgMSBpZiB0aGUgcGFj a2V0cyB3aWxsIGJlIHJlZGlyZWN0ZWQKKyAqIHVwc3RyZWFtLCAwIG90aGVyd2lzZS4KKyAqLwor c3RhdGljIGludCBwY2lfYnJpZGdlX2hhc19hY3NfcmVkaXIoc3RydWN0IHBjaV9kZXYgKmRldikK K3sKKwlpbnQgcG9zOworCXUxNiBjdHJsOworCisJcG9zID0gcGNpX2ZpbmRfZXh0X2NhcGFiaWxp dHkoZGV2LCBQQ0lfRVhUX0NBUF9JRF9BQ1MpOworCWlmICghcG9zKQorCQlyZXR1cm4gMDsKKwor CXBjaV9yZWFkX2NvbmZpZ193b3JkKGRldiwgcG9zICsgUENJX0FDU19DVFJMLCAmY3RybCk7CisK KwlpZiAoY3RybCAmIChQQ0lfQUNTX1JSIHwgUENJX0FDU19DUiB8IFBDSV9BQ1NfRUMpKQorCQly ZXR1cm4gMTsKKworCXJldHVybiAwOworfQorCitzdGF0aWMgdm9pZCBzZXFfYnVmX3ByaW50X2J1 c19kZXZmbihzdHJ1Y3Qgc2VxX2J1ZiAqYnVmLCBzdHJ1Y3QgcGNpX2RldiAqZGV2KQoreworCWlm ICghYnVmKQorCQlyZXR1cm47CisKKwlzZXFfYnVmX3ByaW50ZihidWYsICIlMDR4OiUwMng6JTAy eC4leDsiLCBwY2lfZG9tYWluX25yKGRldi0+YnVzKSwKKwkJICAgICAgIGRldi0+YnVzLT5udW1i ZXIsIFBDSV9TTE9UKGRldi0+ZGV2Zm4pLAorCQkgICAgICAgUENJX0ZVTkMoZGV2LT5kZXZmbikp OworfQorCisvKgorICogRmluZCB0aGUgZGlzdGFuY2UgdGhyb3VnaCB0aGUgbmVhcmVzdCBjb21t b24gdXBzdHJlYW0gYnJpZGdlIGJldHdlZW4KKyAqIHR3byBQQ0kgZGV2aWNlcy4KKyAqCisgKiBJ ZiB0aGUgdHdvIGRldmljZXMgYXJlIHRoZSBzYW1lIGRldmljZSB0aGVuIDAgd2lsbCBiZSByZXR1 cm5lZC4KKyAqCisgKiBJZiB0aGVyZSBhcmUgdHdvIHZpcnR1YWwgZnVuY3Rpb25zIG9mIHRoZSBz YW1lIGRldmljZSBiZWhpbmQgdGhlIHNhbWUKKyAqIGJyaWRnZSBwb3J0IHRoZW4gMiB3aWxsIGJl IHJldHVybmVkIChvbmUgc3RlcCBkb3duIHRvIHRoZSBQQ0llIHN3aXRjaCwKKyAqIHRoZW4gb25l IHN0ZXAgYmFjayB0byB0aGUgc2FtZSBkZXZpY2UpLgorICoKKyAqIEluIHRoZSBjYXNlIHdoZXJl IHR3byBkZXZpY2VzIGFyZSBjb25uZWN0ZWQgdG8gdGhlIHNhbWUgUENJZSBzd2l0Y2gsIHRoZQor ICogdmFsdWUgNCB3aWxsIGJlIHJldHVybmVkLiBUaGlzIGNvcnJlc3BvbmRzIHRvIHRoZSBmb2xs b3dpbmcgUENJIHRyZWU6CisgKgorICogICAgIC0rICBSb290IFBvcnQKKyAqICAgICAgXCsgU3dp dGNoIFVwc3RyZWFtIFBvcnQKKyAqICAgICAgICstKyBTd2l0Y2ggRG93bnN0cmVhbSBQb3J0Cisg KiAgICAgICArIFwtIERldmljZSBBCisgKiAgICAgICBcLSsgU3dpdGNoIERvd25zdHJlYW0gUG9y dAorICogICAgICAgICBcLSBEZXZpY2UgQgorICoKKyAqIFRoZSBkaXN0YW5jZSBpcyA0IGJlY2F1 c2Ugd2UgdHJhdmVyc2UgZnJvbSBEZXZpY2UgQSB0aHJvdWdoIHRoZSBkb3duc3RyZWFtCisgKiBw b3J0IG9mIHRoZSBzd2l0Y2gsIHRvIHRoZSBjb21tb24gdXBzdHJlYW0gcG9ydCwgYmFjayB1cCB0 byB0aGUgc2Vjb25kCisgKiBkb3duc3RyZWFtIHBvcnQgYW5kIHRoZW4gdG8gRGV2aWNlIEIuCisg KgorICogQW55IHR3byBkZXZpY2VzIHRoYXQgZG9uJ3QgaGF2ZSBhIGNvbW1vbiB1cHN0cmVhbSBi cmlkZ2Ugd2lsbCByZXR1cm4gLTEuCisgKiBJbiB0aGlzIHdheSBkZXZpY2VzIG9uIHNlcGFyYXRl IFBDSWUgcm9vdCBwb3J0cyB3aWxsIGJlIHJlamVjdGVkLCB3aGljaAorICogaXMgd2hhdCB3ZSB3 YW50IGZvciBwZWVyLXRvLXBlZXIgc2VlaW5nIGVhY2ggUENJZSByb290IHBvcnQgZGVmaW5lcyBh CisgKiBzZXBhcmF0ZSBoaWVyYXJjaHkgZG9tYWluIGFuZCB0aGVyZSdzIG5vIHdheSB0byBkZXRl cm1pbmUgd2hldGhlciB0aGUgcm9vdAorICogY29tcGxleCBzdXBwb3J0cyBmb3J3YXJkaW5nIGJl dHdlZW4gdGhlbS4KKyAqCisgKiBJbiB0aGUgY2FzZSB3aGVyZSB0d28gZGV2aWNlcyBhcmUgY29u bmVjdGVkIHRvIGRpZmZlcmVudCBQQ0llIHN3aXRjaGVzLAorICogdGhpcyBmdW5jdGlvbiB3aWxs IHN0aWxsIHJldHVybiBhIHBvc2l0aXZlIGRpc3RhbmNlIGFzIGxvbmcgYXMgYm90aAorICogc3dp dGNoZXMgZXZlbnV0YWxseSBoYXZlIGEgY29tbW9uIHVwc3RyZWFtIGJyaWRnZS4gTm90ZSB0aGlz IGNvdmVycworICogdGhlIGNhc2Ugb2YgdXNpbmcgbXVsdGlwbGUgUENJZSBzd2l0Y2hlcyB0byBh Y2hpZXZlIGEgZGVzaXJlZCBsZXZlbCBvZgorICogZmFuLW91dCBmcm9tIGEgcm9vdCBwb3J0LiBU aGUgZXhhY3QgZGlzdGFuY2Ugd2lsbCBiZSBhIGZ1bmN0aW9uIG9mIHRoZQorICogbnVtYmVyIG9m IHN3aXRjaGVzIGJldHdlZW4gRGV2aWNlIEEgYW5kIERldmljZSBCLgorICoKKyAqIElmIGEgYnJp ZGdlIHdoaWNoIGhhcyBhbnkgQUNTIHJlZGlyZWN0aW9uIGJpdHMgc2V0IGlzIGluIHRoZSBwYXRo CisgKiB0aGVuIHRoaXMgZnVuY3Rpb25zIHdpbGwgcmV0dXJuIC0yLiBUaGlzIGlzIHNvIHdlIHJl amVjdCBhbnkKKyAqIGNhc2VzIHdoZXJlIHRoZSBUTFBzIGFyZSBmb3J3YXJkZWQgdXAgaW50byB0 aGUgcm9vdCBjb21wbGV4LgorICogSW4gdGhpcyBjYXNlLCBhIGxpc3Qgb2YgYWxsIGluZnJpbmdp bmcgYnJpZGdlIGFkZHJlc3NlcyB3aWxsIGJlCisgKiBwb3B1bGF0ZWQgaW4gYWNzX2xpc3QgKGFz c3VtaW5nIGl0J3Mgbm9uLW51bGwpIGZvciBwcmludGsgcHVycG9zZXMuCisgKi8KK3N0YXRpYyBp bnQgdXBzdHJlYW1fYnJpZGdlX2Rpc3RhbmNlKHN0cnVjdCBwY2lfZGV2ICphLAorCQkJCSAgICBz dHJ1Y3QgcGNpX2RldiAqYiwKKwkJCQkgICAgc3RydWN0IHNlcV9idWYgKmFjc19saXN0KQorewor CWludCBkaXN0X2EgPSAwOworCWludCBkaXN0X2IgPSAwOworCXN0cnVjdCBwY2lfZGV2ICpiYiA9 IE5VTEw7CisJaW50IGFjc19jbnQgPSAwOworCisJLyoKKwkgKiBOb3RlLCB3ZSBkb24ndCBuZWVk IHRvIHRha2UgcmVmZXJlbmNlcyB0byBkZXZpY2VzIHJldHVybmVkIGJ5CisJICogcGNpX3Vwc3Ry ZWFtX2JyaWRnZSgpIHNlZWluZyB3ZSBob2xkIGEgcmVmZXJlbmNlIHRvIGEgY2hpbGQKKwkgKiBk ZXZpY2Ugd2hpY2ggd2lsbCBhbHJlYWR5IGhvbGQgYSByZWZlcmVuY2UgdG8gdGhlIHVwc3RyZWFt IGJyaWRnZS4KKwkgKi8KKworCXdoaWxlIChhKSB7CisJCWRpc3RfYiA9IDA7CisKKwkJaWYgKHBj aV9icmlkZ2VfaGFzX2Fjc19yZWRpcihhKSkgeworCQkJc2VxX2J1Zl9wcmludF9idXNfZGV2Zm4o YWNzX2xpc3QsIGEpOworCQkJYWNzX2NudCsrOworCQl9CisKKwkJYmIgPSBiOworCisJCXdoaWxl IChiYikgeworCQkJaWYgKGEgPT0gYmIpCisJCQkJZ290byBjaGVja19iX3BhdGhfYWNzOworCisJ CQliYiA9IHBjaV91cHN0cmVhbV9icmlkZ2UoYmIpOworCQkJZGlzdF9iKys7CisJCX0KKworCQlh ID0gcGNpX3Vwc3RyZWFtX2JyaWRnZShhKTsKKwkJZGlzdF9hKys7CisJfQorCisJcmV0dXJuIC0x OworCitjaGVja19iX3BhdGhfYWNzOgorCWJiID0gYjsKKworCXdoaWxlIChiYikgeworCQlpZiAo YSA9PSBiYikKKwkJCWJyZWFrOworCisJCWlmIChwY2lfYnJpZGdlX2hhc19hY3NfcmVkaXIoYmIp KSB7CisJCQlzZXFfYnVmX3ByaW50X2J1c19kZXZmbihhY3NfbGlzdCwgYmIpOworCQkJYWNzX2Nu dCsrOworCQl9CisKKwkJYmIgPSBwY2lfdXBzdHJlYW1fYnJpZGdlKGJiKTsKKwl9CisKKwlpZiAo YWNzX2NudCkKKwkJcmV0dXJuIC0yOworCisJcmV0dXJuIGRpc3RfYSArIGRpc3RfYjsKK30KKwor c3RhdGljIGludCB1cHN0cmVhbV9icmlkZ2VfZGlzdGFuY2Vfd2FybihzdHJ1Y3QgcGNpX2RldiAq cHJvdmlkZXIsCisJCQkJCSBzdHJ1Y3QgcGNpX2RldiAqY2xpZW50KQoreworCXN0cnVjdCBzZXFf YnVmIGFjc19saXN0OworCWludCByZXQ7CisKKwlzZXFfYnVmX2luaXQoJmFjc19saXN0LCBrbWFs bG9jKFBBR0VfU0laRSwgR0ZQX0tFUk5FTCksIFBBR0VfU0laRSk7CisKKwlyZXQgPSB1cHN0cmVh bV9icmlkZ2VfZGlzdGFuY2UocHJvdmlkZXIsIGNsaWVudCwgJmFjc19saXN0KTsKKwlpZiAocmV0 ID09IC0yKSB7CisJCXBjaV93YXJuKGNsaWVudCwgImNhbm5vdCBiZSB1c2VkIGZvciBwZWVyLXRv LXBlZXIgRE1BIGFzIEFDUyByZWRpcmVjdCBpcyBzZXQgYmV0d2VlbiB0aGUgY2xpZW50IGFuZCBw cm92aWRlclxuIik7CisJCS8qIERyb3AgZmluYWwgc2VtaWNvbG9uICovCisJCWFjc19saXN0LmJ1 ZmZlclthY3NfbGlzdC5sZW4tMV0gPSAwOworCQlwY2lfd2FybihjbGllbnQsICJ0byBkaXNhYmxl IEFDUyByZWRpcmVjdCBmb3IgdGhpcyBwYXRoLCBhZGQgdGhlIGtlcm5lbCBwYXJhbWV0ZXI6IHBj aT1kaXNhYmxlX2Fjc19yZWRpcj0lc1xuIiwKKwkJCSBhY3NfbGlzdC5idWZmZXIpOworCisJfSBl bHNlIGlmIChyZXQgPCAwKSB7CisJCXBjaV93YXJuKGNsaWVudCwgImNhbm5vdCBiZSB1c2VkIGZv ciBwZWVyLXRvLXBlZXIgRE1BIGFzIHRoZSBjbGllbnQgYW5kIHByb3ZpZGVyIGRvIG5vdCBzaGFy ZSBhbiB1cHN0cmVhbSBicmlkZ2VcbiIpOworCX0KKworCWtmcmVlKGFjc19saXN0LmJ1ZmZlcik7 CisKKwlyZXR1cm4gcmV0OworfQorCitzdHJ1Y3QgcGNpX3AycGRtYV9jbGllbnQgeworCXN0cnVj dCBsaXN0X2hlYWQgbGlzdDsKKwlzdHJ1Y3QgcGNpX2RldiAqY2xpZW50OworCXN0cnVjdCBwY2lf ZGV2ICpwcm92aWRlcjsKK307CisKKy8qKgorICogcGNpX3AycGRtYV9hZGRfY2xpZW50IC0gYWxs b2NhdGUgYSBuZXcgZWxlbWVudCBpbiBhIGNsaWVudCBkZXZpY2UgbGlzdAorICogQGhlYWQ6IGxp c3QgaGVhZCBvZiBwMnBkbWEgY2xpZW50cworICogQGRldjogZGV2aWNlIHRvIGFkZCB0byB0aGUg bGlzdAorICoKKyAqIFRoaXMgYWRkcyBAZGV2IHRvIGEgbGlzdCBvZiBjbGllbnRzIHVzZWQgYnkg YSBwMnBkbWEgZGV2aWNlLgorICogVGhpcyBsaXN0IHNob3VsZCBiZSBwYXNzZWQgdG8gcGNpX3Ay cG1lbV9maW5kKCkuIE9uY2UgcGNpX3AycG1lbV9maW5kKCkgaGFzCisgKiBiZWVuIGNhbGxlZCBz dWNjZXNzZnVsbHksIHRoZSBsaXN0IHdpbGwgYmUgYm91bmQgdG8gYSBzcGVjaWZpYyBwMnBkbWEK KyAqIGRldmljZSBhbmQgbmV3IGNsaWVudHMgY2FuIG9ubHkgYmUgYWRkZWQgdG8gdGhlIGxpc3Qg aWYgdGhleSBhcmUKKyAqIHN1cHBvcnRlZCBieSB0aGF0IHAycGRtYSBkZXZpY2UuCisgKgorICog VGhlIGNhbGxlciBpcyBleHBlY3RlZCB0byBoYXZlIGEgbG9jayB3aGljaCBwcm90ZWN0cyBAaGVh ZCBhcyBuZWNlc3NhcnkKKyAqIHNvIHRoYXQgbm9uZSBvZiB0aGUgcGNpX3AycCBmdW5jdGlvbnMg Y2FuIGJlIGNhbGxlZCBjb25jdXJyZW50bHkKKyAqIG9uIHRoYXQgbGlzdC4KKyAqCisgKiBSZXR1 cm5zIDAgaWYgdGhlIGNsaWVudCB3YXMgc3VjY2Vzc2Z1bGx5IGFkZGVkLgorICovCitpbnQgcGNp X3AycGRtYV9hZGRfY2xpZW50KHN0cnVjdCBsaXN0X2hlYWQgKmhlYWQsIHN0cnVjdCBkZXZpY2Ug KmRldikKK3sKKwlzdHJ1Y3QgcGNpX3AycGRtYV9jbGllbnQgKml0ZW0sICpuZXdfaXRlbTsKKwlz dHJ1Y3QgcGNpX2RldiAqcHJvdmlkZXIgPSBOVUxMOworCXN0cnVjdCBwY2lfZGV2ICpjbGllbnQ7 CisJaW50IHJldDsKKworCWlmIChJU19FTkFCTEVEKENPTkZJR19ETUFfVklSVF9PUFMpICYmIGRl di0+ZG1hX29wcyA9PSAmZG1hX3ZpcnRfb3BzKSB7CisJCWRldl93YXJuKGRldiwgImNhbm5vdCBi ZSB1c2VkIGZvciBwZWVyLXRvLXBlZXIgRE1BIGJlY2F1c2UgdGhlIGRyaXZlciBtYWtlcyB1c2Ug b2YgZG1hX3ZpcnRfb3BzXG4iKTsKKwkJcmV0dXJuIC1FTk9ERVY7CisJfQorCisJY2xpZW50ID0g ZmluZF9wYXJlbnRfcGNpX2RldihkZXYpOworCWlmICghY2xpZW50KSB7CisJCWRldl93YXJuKGRl diwgImNhbm5vdCBiZSB1c2VkIGZvciBwZWVyLXRvLXBlZXIgRE1BIGFzIGl0IGlzIG5vdCBhIFBD SSBkZXZpY2VcbiIpOworCQlyZXR1cm4gLUVOT0RFVjsKKwl9CisKKwlpdGVtID0gbGlzdF9maXJz dF9lbnRyeV9vcl9udWxsKGhlYWQsIHN0cnVjdCBwY2lfcDJwZG1hX2NsaWVudCwgbGlzdCk7CisJ aWYgKGl0ZW0gJiYgaXRlbS0+cHJvdmlkZXIpIHsKKwkJcHJvdmlkZXIgPSBpdGVtLT5wcm92aWRl cjsKKworCQlyZXQgPSB1cHN0cmVhbV9icmlkZ2VfZGlzdGFuY2Vfd2Fybihwcm92aWRlciwgY2xp ZW50KTsKKwkJaWYgKHJldCA8IDApIHsKKwkJCXJldCA9IC1FWERFVjsKKwkJCWdvdG8gcHV0X2Ns aWVudDsKKwkJfQorCX0KKworCW5ld19pdGVtID0ga3phbGxvYyhzaXplb2YoKm5ld19pdGVtKSwg R0ZQX0tFUk5FTCk7CisJaWYgKCFuZXdfaXRlbSkgeworCQlyZXQgPSAtRU5PTUVNOworCQlnb3Rv IHB1dF9jbGllbnQ7CisJfQorCisJbmV3X2l0ZW0tPmNsaWVudCA9IGNsaWVudDsKKwluZXdfaXRl bS0+cHJvdmlkZXIgPSBwY2lfZGV2X2dldChwcm92aWRlcik7CisKKwlsaXN0X2FkZF90YWlsKCZu ZXdfaXRlbS0+bGlzdCwgaGVhZCk7CisKKwlyZXR1cm4gMDsKKworcHV0X2NsaWVudDoKKwlwY2lf ZGV2X3B1dChjbGllbnQpOworCXJldHVybiByZXQ7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lf cDJwZG1hX2FkZF9jbGllbnQpOworCitzdGF0aWMgdm9pZCBwY2lfcDJwZG1hX2NsaWVudF9mcmVl KHN0cnVjdCBwY2lfcDJwZG1hX2NsaWVudCAqaXRlbSkKK3sKKwlsaXN0X2RlbCgmaXRlbS0+bGlz dCk7CisJcGNpX2Rldl9wdXQoaXRlbS0+Y2xpZW50KTsKKwlwY2lfZGV2X3B1dChpdGVtLT5wcm92 aWRlcik7CisJa2ZyZWUoaXRlbSk7Cit9CisKKy8qKgorICogcGNpX3AycGRtYV9yZW1vdmVfY2xp ZW50IC0gcmVtb3ZlIGFuZCBmcmVlIGEgcDJwZG1hIGNsaWVudAorICogQGhlYWQ6IGxpc3QgaGVh ZCBvZiBwMnBkbWEgY2xpZW50cworICogQGRldjogZGV2aWNlIHRvIHJlbW92ZSBmcm9tIHRoZSBs aXN0CisgKgorICogVGhpcyByZW1vdmVzIEBkZXYgZnJvbSBhIGxpc3Qgb2YgY2xpZW50cyB1c2Vk IGJ5IGEgcDJwZG1hIGRldmljZS4KKyAqIFRoZSBjYWxsZXIgaXMgZXhwZWN0ZWQgdG8gaGF2ZSBh IGxvY2sgd2hpY2ggcHJvdGVjdHMgQGhlYWQgYXMgbmVjZXNzYXJ5CisgKiBzbyB0aGF0IG5vbmUg b2YgdGhlIHBjaV9wMnAgZnVuY3Rpb25zIGNhbiBiZSBjYWxsZWQgY29uY3VycmVudGx5CisgKiBv biB0aGF0IGxpc3QuCisgKi8KK3ZvaWQgcGNpX3AycGRtYV9yZW1vdmVfY2xpZW50KHN0cnVjdCBs aXN0X2hlYWQgKmhlYWQsIHN0cnVjdCBkZXZpY2UgKmRldikKK3sKKwlzdHJ1Y3QgcGNpX3AycGRt YV9jbGllbnQgKnBvcywgKnRtcDsKKwlzdHJ1Y3QgcGNpX2RldiAqcGRldjsKKworCXBkZXYgPSBm aW5kX3BhcmVudF9wY2lfZGV2KGRldik7CisJaWYgKCFwZGV2KQorCQlyZXR1cm47CisKKwlsaXN0 X2Zvcl9lYWNoX2VudHJ5X3NhZmUocG9zLCB0bXAsIGhlYWQsIGxpc3QpIHsKKwkJaWYgKHBvcy0+ Y2xpZW50ICE9IHBkZXYpCisJCQljb250aW51ZTsKKworCQlwY2lfcDJwZG1hX2NsaWVudF9mcmVl KHBvcyk7CisJfQorCisJcGNpX2Rldl9wdXQocGRldik7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChw Y2lfcDJwZG1hX3JlbW92ZV9jbGllbnQpOworCisvKioKKyAqIHBjaV9wMnBkbWFfY2xpZW50X2xp c3RfZnJlZSAtIGZyZWUgYW4gZW50aXJlIGxpc3Qgb2YgcDJwZG1hIGNsaWVudHMKKyAqIEBoZWFk OiBsaXN0IGhlYWQgb2YgcDJwZG1hIGNsaWVudHMKKyAqCisgKiBUaGlzIHJlbW92ZXMgYWxsIGRl dmljZXMgaW4gYSBsaXN0IG9mIGNsaWVudHMgdXNlZCBieSBhIHAycGRtYSBkZXZpY2UuCisgKiBU aGUgY2FsbGVyIGlzIGV4cGVjdGVkIHRvIGhhdmUgYSBsb2NrIHdoaWNoIHByb3RlY3RzIEBoZWFk IGFzIG5lY2Vzc2FyeQorICogc28gdGhhdCBub25lIG9mIHRoZSBwY2lfcDJwZG1hIGZ1bmN0aW9u cyBjYW4gYmUgY2FsbGVkIGNvbmN1cnJlbnRseQorICogb24gdGhhdCBsaXN0LgorICovCit2b2lk IHBjaV9wMnBkbWFfY2xpZW50X2xpc3RfZnJlZShzdHJ1Y3QgbGlzdF9oZWFkICpoZWFkKQorewor CXN0cnVjdCBwY2lfcDJwZG1hX2NsaWVudCAqcG9zLCAqdG1wOworCisJbGlzdF9mb3JfZWFjaF9l bnRyeV9zYWZlKHBvcywgdG1wLCBoZWFkLCBsaXN0KQorCQlwY2lfcDJwZG1hX2NsaWVudF9mcmVl KHBvcyk7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lfcDJwZG1hX2NsaWVudF9saXN0X2ZyZWUp OworCisvKioKKyAqIHBjaV9wMnBkbWFfZGlzdGFuY2UgLSBEZXRlcm1pdmUgdGhlIGN1bXVsYXRp dmUgZGlzdGFuY2UgYmV0d2VlbgorICoJYSBwMnBkbWEgcHJvdmlkZXIgYW5kIHRoZSBjbGllbnRz IGluIHVzZS4KKyAqIEBwcm92aWRlcjogcDJwZG1hIHByb3ZpZGVyIHRvIGNoZWNrIGFnYWluc3Qg dGhlIGNsaWVudCBsaXN0CisgKiBAY2xpZW50czogbGlzdCBvZiBkZXZpY2VzIHRvIGNoZWNrIChO VUxMLXRlcm1pbmF0ZWQpCisgKiBAdmVyYm9zZTogaWYgdHJ1ZSwgcHJpbnQgd2FybmluZ3MgZm9y IGRldmljZXMgd2hlbiB3ZSByZXR1cm4gLTEKKyAqCisgKiBSZXR1cm5zIC0xIGlmIGFueSBvZiB0 aGUgY2xpZW50cyBhcmUgbm90IGNvbXBhdGlibGUgKGJlaGluZCB0aGUgc2FtZQorICogcm9vdCBw b3J0IGFzIHRoZSBwcm92aWRlciksIG90aGVyd2lzZSByZXR1cm5zIGEgcG9zaXRpdmUgbnVtYmVy IHdoZXJlCisgKiB0aGUgbG93ZXIgbnVtYmVyIGlzIHRoZSBwcmVmZXJyYWJsZSBjaG9pY2UuIChJ ZiB0aGVyZSdzIG9uZSBjbGllbnQKKyAqIHRoYXQncyB0aGUgc2FtZSBhcyB0aGUgcHJvdmlkZXIg aXQgd2lsbCByZXR1cm4gMCwgd2hpY2ggaXMgYmVzdCBjaG9pY2UpLgorICoKKyAqIEZvciBub3cs ICJjb21wYXRpYmxlIiBtZWFucyB0aGUgcHJvdmlkZXIgYW5kIHRoZSBjbGllbnRzIGFyZSBhbGwg YmVoaW5kCisgKiB0aGUgc2FtZSBQQ0kgcm9vdCBwb3J0LiBUaGlzIGN1dHMgb3V0IGNhc2VzIHRo YXQgbWF5IHdvcmsgYnV0IGlzIHNhZmVzdAorICogZm9yIHRoZSB1c2VyLiBGdXR1cmUgd29yayBj YW4gZXhwYW5kIHRoaXMgdG8gd2hpdGUtbGlzdCByb290IGNvbXBsZXhlcyB0aGF0CisgKiBjYW4g c2FmZWx5IGZvcndhcmQgYmV0d2VlbiBlYWNoIHBvcnRzLgorICovCitpbnQgcGNpX3AycGRtYV9k aXN0YW5jZShzdHJ1Y3QgcGNpX2RldiAqcHJvdmlkZXIsIHN0cnVjdCBsaXN0X2hlYWQgKmNsaWVu dHMsCisJCQlib29sIHZlcmJvc2UpCit7CisJc3RydWN0IHBjaV9wMnBkbWFfY2xpZW50ICpwb3M7 CisJaW50IHJldDsKKwlpbnQgZGlzdGFuY2UgPSAwOworCWJvb2wgbm90X3N1cHBvcnRlZCA9IGZh bHNlOworCisJaWYgKGxpc3RfZW1wdHkoY2xpZW50cykpCisJCXJldHVybiAtMTsKKworCWxpc3Rf Zm9yX2VhY2hfZW50cnkocG9zLCBjbGllbnRzLCBsaXN0KSB7CisJCWlmICh2ZXJib3NlKQorCQkJ cmV0ID0gdXBzdHJlYW1fYnJpZGdlX2Rpc3RhbmNlX3dhcm4ocHJvdmlkZXIsCisJCQkJCQkJICAg IHBvcy0+Y2xpZW50KTsKKwkJZWxzZQorCQkJcmV0ID0gdXBzdHJlYW1fYnJpZGdlX2Rpc3RhbmNl KHByb3ZpZGVyLCBwb3MtPmNsaWVudCwKKwkJCQkJCSAgICAgICBOVUxMKTsKKworCQlpZiAocmV0 IDwgMCkKKwkJCW5vdF9zdXBwb3J0ZWQgPSB0cnVlOworCisJCWlmIChub3Rfc3VwcG9ydGVkICYm ICF2ZXJib3NlKQorCQkJYnJlYWs7CisKKwkJZGlzdGFuY2UgKz0gcmV0OworCX0KKworCWlmIChu b3Rfc3VwcG9ydGVkKQorCQlyZXR1cm4gLTE7CisKKwlyZXR1cm4gZGlzdGFuY2U7Cit9CitFWFBP UlRfU1lNQk9MX0dQTChwY2lfcDJwZG1hX2Rpc3RhbmNlKTsKKworLyoqCisgKiBwY2lfcDJwZG1h X2Fzc2lnbl9wcm92aWRlciAtIENoZWNrIGNvbXBhdGliaWx5IChhcyBwZXIgcGNpX3AycGRtYV9k aXN0YW5jZSkKKyAqCWFuZCBhc3NpZ24gYSBwcm92aWRlciB0byBhIGxpc3Qgb2YgY2xpZW50cwor ICogQHByb3ZpZGVyOiBwMnBkbWEgcHJvdmlkZXIgdG8gYXNzaWduIHRvIHRoZSBjbGllbnQgbGlz dAorICogQGNsaWVudHM6IGxpc3Qgb2YgZGV2aWNlcyB0byBjaGVjayAoTlVMTC10ZXJtaW5hdGVk KQorICoKKyAqIFJldHVybnMgZmFsc2UgaWYgYW55IG9mIHRoZSBjbGllbnRzIGFyZSBub3QgY29t cGF0aWJsZSwgdHJ1ZSBpZiB0aGUKKyAqIHByb3ZpZGVyIHdhcyBzdWNjZXNzZnVsbHkgYXNzaWdu ZWQgdG8gdGhlIGNsaWVudHMuCisgKi8KK2Jvb2wgcGNpX3AycGRtYV9hc3NpZ25fcHJvdmlkZXIo c3RydWN0IHBjaV9kZXYgKnByb3ZpZGVyLAorCQkJCXN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMp Cit7CisJc3RydWN0IHBjaV9wMnBkbWFfY2xpZW50ICpwb3M7CisKKwlpZiAocGNpX3AycGRtYV9k aXN0YW5jZShwcm92aWRlciwgY2xpZW50cywgdHJ1ZSkgPCAwKQorCQlyZXR1cm4gZmFsc2U7CisK KwlsaXN0X2Zvcl9lYWNoX2VudHJ5KHBvcywgY2xpZW50cywgbGlzdCkKKwkJcG9zLT5wcm92aWRl ciA9IHByb3ZpZGVyOworCisJcmV0dXJuIHRydWU7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lf cDJwZG1hX2Fzc2lnbl9wcm92aWRlcik7CisKKy8qKgorICogcGNpX2hhc19wMnBtZW0gLSBjaGVj ayBpZiBhIGdpdmVuIFBDSSBkZXZpY2UgaGFzIHB1Ymxpc2hlZCBhbnkgcDJwbWVtCisgKiBAcGRl djogUENJIGRldmljZSB0byBjaGVjaworICovCitib29sIHBjaV9oYXNfcDJwbWVtKHN0cnVjdCBw Y2lfZGV2ICpwZGV2KQoreworCXJldHVybiBwZGV2LT5wMnBkbWEgJiYgcGRldi0+cDJwZG1hLT5w MnBtZW1fcHVibGlzaGVkOworfQorRVhQT1JUX1NZTUJPTF9HUEwocGNpX2hhc19wMnBtZW0pOwor CisvKioKKyAqIHBjaV9wMnBtZW1fZmluZCAtIGZpbmQgYSBwZWVyLXRvLXBlZXIgRE1BIG1lbW9y eSBkZXZpY2UgY29tcGF0aWJsZSB3aXRoCisgKgl0aGUgc3BlY2lmaWVkIGxpc3Qgb2YgY2xpZW50 cyBhbmQgc2hvcnRlc3QgZGlzdGFuY2UgKGFzIGRldGVybWluZWQKKyAqCWJ5IHBjaV9wMnBtZW1f ZG1hKCkpCisgKiBAY2xpZW50czogbGlzdCBvZiBkZXZpY2VzIHRvIGNoZWNrIChOVUxMLXRlcm1p bmF0ZWQpCisgKgorICogSWYgbXVsdGlwbGUgZGV2aWNlcyBhcmUgYmVoaW5kIHRoZSBzYW1lIHN3 aXRjaCwgdGhlIG9uZSAiY2xvc2VzdCIgdG8gdGhlCisgKiBjbGllbnQgZGV2aWNlcyBpbiB1c2Ug d2lsbCBiZSBjaG9zZW4gZmlyc3QuIChTbyBpZiBvbmUgb2YgdGhlIHByb3ZpZGVycyBhcmUKKyAq IHRoZSBzYW1lIGFzIG9uZSBvZiB0aGUgY2xpZW50cywgdGhhdCBwcm92aWRlciB3aWxsIGJlIHVz ZWQgYWhlYWQgb2YgYW55CisgKiBvdGhlciBwcm92aWRlcnMgdGhhdCBhcmUgdW5yZWxhdGVkKS4g SWYgbXVsdGlwbGUgcHJvdmlkZXJzIGFyZSBhbiBlcXVhbAorICogZGlzdGFuY2UgYXdheSwgb25l IHdpbGwgYmUgY2hvc2VuIGF0IHJhbmRvbS4KKyAqCisgKiBSZXR1cm5zIGEgcG9pbnRlciB0byB0 aGUgUENJIGRldmljZSB3aXRoIGEgcmVmZXJlbmNlIHRha2VuICh1c2UgcGNpX2Rldl9wdXQKKyAq IHRvIHJldHVybiB0aGUgcmVmZXJlbmNlKSBvciBOVUxMIGlmIG5vIGNvbXBhdGlibGUgZGV2aWNl IGlzIGZvdW5kLiBUaGUKKyAqIGZvdW5kIHByb3ZpZGVyIHdpbGwgYWxzbyBiZSBhc3NpZ25lZCB0 byB0aGUgY2xpZW50IGxpc3QuCisgKi8KK3N0cnVjdCBwY2lfZGV2ICpwY2lfcDJwbWVtX2ZpbmQo c3RydWN0IGxpc3RfaGVhZCAqY2xpZW50cykKK3sKKwlzdHJ1Y3QgcGNpX2RldiAqcGRldiA9IE5V TEw7CisJc3RydWN0IHBjaV9wMnBkbWFfY2xpZW50ICpwb3M7CisJaW50IGRpc3RhbmNlOworCWlu dCBjbG9zZXN0X2Rpc3RhbmNlID0gSU5UX01BWDsKKwlzdHJ1Y3QgcGNpX2RldiAqKmNsb3Nlc3Rf cGRldnM7CisJaW50IGRldl9jbnQgPSAwOworCWNvbnN0IGludCBtYXhfZGV2cyA9IFBBR0VfU0la RSAvIHNpemVvZigqY2xvc2VzdF9wZGV2cyk7CisJaW50IGk7CisKKwljbG9zZXN0X3BkZXZzID0g a21hbGxvYyhQQUdFX1NJWkUsIEdGUF9LRVJORUwpOworCisJd2hpbGUgKChwZGV2ID0gcGNpX2dl dF9kZXZpY2UoUENJX0FOWV9JRCwgUENJX0FOWV9JRCwgcGRldikpKSB7CisJCWlmICghcGNpX2hh c19wMnBtZW0ocGRldikpCisJCQljb250aW51ZTsKKworCQlkaXN0YW5jZSA9IHBjaV9wMnBkbWFf ZGlzdGFuY2UocGRldiwgY2xpZW50cywgZmFsc2UpOworCQlpZiAoZGlzdGFuY2UgPCAwIHx8IGRp c3RhbmNlID4gY2xvc2VzdF9kaXN0YW5jZSkKKwkJCWNvbnRpbnVlOworCisJCWlmIChkaXN0YW5j ZSA9PSBjbG9zZXN0X2Rpc3RhbmNlICYmIGRldl9jbnQgPj0gbWF4X2RldnMpCisJCQljb250aW51 ZTsKKworCQlpZiAoZGlzdGFuY2UgPCBjbG9zZXN0X2Rpc3RhbmNlKSB7CisJCQlmb3IgKGkgPSAw OyBpIDwgZGV2X2NudDsgaSsrKQorCQkJCXBjaV9kZXZfcHV0KGNsb3Nlc3RfcGRldnNbaV0pOwor CisJCQlkZXZfY250ID0gMDsKKwkJCWNsb3Nlc3RfZGlzdGFuY2UgPSBkaXN0YW5jZTsKKwkJfQor CisJCWNsb3Nlc3RfcGRldnNbZGV2X2NudCsrXSA9IHBjaV9kZXZfZ2V0KHBkZXYpOworCX0KKwor CWlmIChkZXZfY250KQorCQlwZGV2ID0gcGNpX2Rldl9nZXQoY2xvc2VzdF9wZGV2c1twcmFuZG9t X3UzMl9tYXgoZGV2X2NudCldKTsKKworCWZvciAoaSA9IDA7IGkgPCBkZXZfY250OyBpKyspCisJ CXBjaV9kZXZfcHV0KGNsb3Nlc3RfcGRldnNbaV0pOworCisJaWYgKHBkZXYpCisJCWxpc3RfZm9y X2VhY2hfZW50cnkocG9zLCBjbGllbnRzLCBsaXN0KQorCQkJcG9zLT5wcm92aWRlciA9IHBkZXY7 CisKKwlrZnJlZShjbG9zZXN0X3BkZXZzKTsKKwlyZXR1cm4gcGRldjsKK30KK0VYUE9SVF9TWU1C T0xfR1BMKHBjaV9wMnBtZW1fZmluZCk7CisKKy8qKgorICogcGNpX2FsbG9jX3AycF9tZW0gLSBh bGxvY2F0ZSBwZWVyLXRvLXBlZXIgRE1BIG1lbW9yeQorICogQHBkZXY6IHRoZSBkZXZpY2UgdG8g YWxsb2NhdGUgbWVtb3J5IGZyb20KKyAqIEBzaXplOiBudW1iZXIgb2YgYnl0ZXMgdG8gYWxsb2Nh dGUKKyAqCisgKiBSZXR1cm5zIHRoZSBhbGxvY2F0ZWQgbWVtb3J5IG9yIE5VTEwgb24gZXJyb3Iu CisgKi8KK3ZvaWQgKnBjaV9hbGxvY19wMnBtZW0oc3RydWN0IHBjaV9kZXYgKnBkZXYsIHNpemVf dCBzaXplKQoreworCXZvaWQgKnJldDsKKworCWlmICh1bmxpa2VseSghcGRldi0+cDJwZG1hKSkK KwkJcmV0dXJuIE5VTEw7CisKKwlpZiAodW5saWtlbHkoIXBlcmNwdV9yZWZfdHJ5Z2V0X2xpdmUo JnBkZXYtPnAycGRtYS0+ZGV2bWFwX3JlZikpKQorCQlyZXR1cm4gTlVMTDsKKworCXJldCA9ICh2 b2lkICopZ2VuX3Bvb2xfYWxsb2MocGRldi0+cDJwZG1hLT5wb29sLCBzaXplKTsKKworCWlmICh1 bmxpa2VseSghcmV0KSkKKwkJcGVyY3B1X3JlZl9wdXQoJnBkZXYtPnAycGRtYS0+ZGV2bWFwX3Jl Zik7CisKKwlyZXR1cm4gcmV0OworfQorRVhQT1JUX1NZTUJPTF9HUEwocGNpX2FsbG9jX3AycG1l bSk7CisKKy8qKgorICogcGNpX2ZyZWVfcDJwbWVtIC0gYWxsb2NhdGUgcGVlci10by1wZWVyIERN QSBtZW1vcnkKKyAqIEBwZGV2OiB0aGUgZGV2aWNlIHRoZSBtZW1vcnkgd2FzIGFsbG9jYXRlZCBm cm9tCisgKiBAYWRkcjogYWRkcmVzcyBvZiB0aGUgbWVtb3J5IHRoYXQgd2FzIGFsbG9jYXRlZAor ICogQHNpemU6IG51bWJlciBvZiBieXRlcyB0aGF0IHdhcyBhbGxvY2F0ZWQKKyAqLwordm9pZCBw Y2lfZnJlZV9wMnBtZW0oc3RydWN0IHBjaV9kZXYgKnBkZXYsIHZvaWQgKmFkZHIsIHNpemVfdCBz aXplKQoreworCWdlbl9wb29sX2ZyZWUocGRldi0+cDJwZG1hLT5wb29sLCAodWludHB0cl90KWFk ZHIsIHNpemUpOworCXBlcmNwdV9yZWZfcHV0KCZwZGV2LT5wMnBkbWEtPmRldm1hcF9yZWYpOwor fQorRVhQT1JUX1NZTUJPTF9HUEwocGNpX2ZyZWVfcDJwbWVtKTsKKworLyoqCisgKiBwY2lfdmly dF90b19idXMgLSByZXR1cm4gdGhlIFBDSSBidXMgYWRkcmVzcyBmb3IgYSBnaXZlbiB2aXJ0dWFs CisgKglhZGRyZXNzIG9idGFpbmVkIHdpdGggcGNpX2FsbG9jX3AycG1lbSgpCisgKiBAcGRldjog dGhlIGRldmljZSB0aGUgbWVtb3J5IHdhcyBhbGxvY2F0ZWQgZnJvbQorICogQGFkZHI6IGFkZHJl c3Mgb2YgdGhlIG1lbW9yeSB0aGF0IHdhcyBhbGxvY2F0ZWQKKyAqLworcGNpX2J1c19hZGRyX3Qg cGNpX3AycG1lbV92aXJ0X3RvX2J1cyhzdHJ1Y3QgcGNpX2RldiAqcGRldiwgdm9pZCAqYWRkcikK K3sKKwlpZiAoIWFkZHIpCisJCXJldHVybiAwOworCWlmICghcGRldi0+cDJwZG1hKQorCQlyZXR1 cm4gMDsKKworCS8qCisJICogTm90ZTogd2hlbiB3ZSBhZGRlZCB0aGUgbWVtb3J5IHRvIHRoZSBw b29sIHdlIHVzZWQgdGhlIFBDSQorCSAqIGJ1cyBhZGRyZXNzIGFzIHRoZSBwaHlzaWNhbCBhZGRy ZXNzLiBTbyBnZW5fcG9vbF92aXJ0X3RvX3BoeXMoKQorCSAqIGFjdHVhbGx5IHJldHVybnMgdGhl IGJ1cyBhZGRyZXNzIGRlc3BpdGUgdGhlIG1pc2xlYWRpbmcgbmFtZS4KKwkgKi8KKwlyZXR1cm4g Z2VuX3Bvb2xfdmlydF90b19waHlzKHBkZXYtPnAycGRtYS0+cG9vbCwgKHVuc2lnbmVkIGxvbmcp YWRkcik7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lfcDJwbWVtX3ZpcnRfdG9fYnVzKTsKKwor LyoqCisgKiBwY2lfcDJwbWVtX2FsbG9jX3NnbCAtIGFsbG9jYXRlIHBlZXItdG8tcGVlciBETUEg bWVtb3J5IGluIGEgc2NhdHRlcmxpc3QKKyAqIEBwZGV2OiB0aGUgZGV2aWNlIHRvIGFsbG9jYXRl IG1lbW9yeSBmcm9tCisgKiBAc2dsOiB0aGUgYWxsb2NhdGVkIHNjYXR0ZXJsaXN0CisgKiBAbmVu dHM6IHRoZSBudW1iZXIgb2YgU0cgZW50cmllcyBpbiB0aGUgbGlzdAorICogQGxlbmd0aDogbnVt YmVyIG9mIGJ5dGVzIHRvIGFsbG9jYXRlCisgKgorICogUmV0dXJucyAwIG9uIHN1Y2Nlc3MKKyAq Lworc3RydWN0IHNjYXR0ZXJsaXN0ICpwY2lfcDJwbWVtX2FsbG9jX3NnbChzdHJ1Y3QgcGNpX2Rl diAqcGRldiwKKwkJCQkJIHVuc2lnbmVkIGludCAqbmVudHMsIHUzMiBsZW5ndGgpCit7CisJc3Ry dWN0IHNjYXR0ZXJsaXN0ICpzZzsKKwl2b2lkICphZGRyOworCisJc2cgPSBremFsbG9jKHNpemVv Zigqc2cpLCBHRlBfS0VSTkVMKTsKKwlpZiAoIXNnKQorCQlyZXR1cm4gTlVMTDsKKworCXNnX2lu aXRfdGFibGUoc2csIDEpOworCisJYWRkciA9IHBjaV9hbGxvY19wMnBtZW0ocGRldiwgbGVuZ3Ro KTsKKwlpZiAoIWFkZHIpCisJCWdvdG8gb3V0X2ZyZWVfc2c7CisKKwlzZ19zZXRfYnVmKHNnLCBh ZGRyLCBsZW5ndGgpOworCSpuZW50cyA9IDE7CisJcmV0dXJuIHNnOworCitvdXRfZnJlZV9zZzoK KwlrZnJlZShzZyk7CisJcmV0dXJuIE5VTEw7Cit9CitFWFBPUlRfU1lNQk9MX0dQTChwY2lfcDJw bWVtX2FsbG9jX3NnbCk7CisKKy8qKgorICogcGNpX3AycG1lbV9mcmVlX3NnbCAtIGZyZWUgYSBz Y2F0dGVybGlzdCBhbGxvY2F0ZWQgYnkgcGNpX3AycG1lbV9hbGxvY19zZ2woKQorICogQHBkZXY6 IHRoZSBkZXZpY2UgdG8gYWxsb2NhdGUgbWVtb3J5IGZyb20KKyAqIEBzZ2w6IHRoZSBhbGxvY2F0 ZWQgc2NhdHRlcmxpc3QKKyAqIEBuZW50czogdGhlIG51bWJlciBvZiBTRyBlbnRyaWVzIGluIHRo ZSBsaXN0CisgKi8KK3ZvaWQgcGNpX3AycG1lbV9mcmVlX3NnbChzdHJ1Y3QgcGNpX2RldiAqcGRl diwgc3RydWN0IHNjYXR0ZXJsaXN0ICpzZ2wpCit7CisJc3RydWN0IHNjYXR0ZXJsaXN0ICpzZzsK KwlpbnQgY291bnQ7CisKKwlmb3JfZWFjaF9zZyhzZ2wsIHNnLCBJTlRfTUFYLCBjb3VudCkgewor CQlpZiAoIXNnKQorCQkJYnJlYWs7CisKKwkJcGNpX2ZyZWVfcDJwbWVtKHBkZXYsIHNnX3ZpcnQo c2cpLCBzZy0+bGVuZ3RoKTsKKwl9CisJa2ZyZWUoc2dsKTsKK30KK0VYUE9SVF9TWU1CT0xfR1BM KHBjaV9wMnBtZW1fZnJlZV9zZ2wpOworCisvKioKKyAqIHBjaV9wMnBtZW1fcHVibGlzaCAtIHB1 Ymxpc2ggdGhlIHBlZXItdG8tcGVlciBETUEgbWVtb3J5IGZvciB1c2UgYnkKKyAqCW90aGVyIGRl dmljZXMgd2l0aCBwY2lfcDJwbWVtX2ZpbmQoKQorICogQHBkZXY6IHRoZSBkZXZpY2Ugd2l0aCBw ZWVyLXRvLXBlZXIgRE1BIG1lbW9yeSB0byBwdWJsaXNoCisgKiBAcHVibGlzaDogc2V0IHRvIHRy dWUgdG8gcHVibGlzaCB0aGUgbWVtb3J5LCBmYWxzZSB0byB1bnB1Ymxpc2ggaXQKKyAqCisgKiBQ dWJsaXNoZWQgbWVtb3J5IGNhbiBiZSB1c2VkIGJ5IG90aGVyIFBDSSBkZXZpY2UgZHJpdmVycyBm b3IKKyAqIHBlZXItMi1wZWVyIERNQSBvcGVyYXRpb25zLiBOb24tcHVibGlzaGVkIG1lbW9yeSBp cyByZXNlcnZlZCBmb3IKKyAqIGV4bHVzaXZlIHVzZSBvZiB0aGUgZGV2aWNlIGRyaXZlciB0aGF0 IHJlZ2lzdGVycyB0aGUgcGVlci10by1wZWVyCisgKiBtZW1vcnkuCisgKi8KK3ZvaWQgcGNpX3Ay cG1lbV9wdWJsaXNoKHN0cnVjdCBwY2lfZGV2ICpwZGV2LCBib29sIHB1Ymxpc2gpCit7CisJaWYg KHB1Ymxpc2ggJiYgIXBkZXYtPnAycGRtYSkKKwkJcmV0dXJuOworCisJcGRldi0+cDJwZG1hLT5w MnBtZW1fcHVibGlzaGVkID0gcHVibGlzaDsKK30KK0VYUE9SVF9TWU1CT0xfR1BMKHBjaV9wMnBt ZW1fcHVibGlzaCk7CmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21lbXJlbWFwLmggYi9pbmNs dWRlL2xpbnV4L21lbXJlbWFwLmgKaW5kZXggZjkxZjllNzYzNTU3Li45NTUzMzcwZWJkYWQgMTAw NjQ0Ci0tLSBhL2luY2x1ZGUvbGludXgvbWVtcmVtYXAuaAorKysgYi9pbmNsdWRlL2xpbnV4L21l bXJlbWFwLmgKQEAgLTUzLDExICs1MywxNiBAQCBzdHJ1Y3Qgdm1lbV9hbHRtYXAgewogICogd2Fr ZXVwIGV2ZW50IHdoZW5ldmVyIGEgcGFnZSBpcyB1bnBpbm5lZCBhbmQgYmVjb21lcyBpZGxlLiBU aGlzCiAgKiB3YWtldXAgaXMgdXNlZCB0byBjb29yZGluYXRlIHBoeXNpY2FsIGFkZHJlc3Mgc3Bh Y2UgbWFuYWdlbWVudCAoZXg6CiAgKiBmcyB0cnVuY2F0ZS9ob2xlIHB1bmNoKSB2cyBwaW5uZWQg cGFnZXMgKGV4OiBkZXZpY2UgZG1hKS4KKyAqCisgKiBNRU1PUllfREVWSUNFX1BDSV9QMlBETUE6 CisgKiBEZXZpY2UgbWVtb3J5IHJlc2lkaW5nIGluIGEgUENJIEJBUiBpbnRlbmRlZCBmb3IgdXNl IHdpdGggUGVlci10by1QZWVyCisgKiB0cmFuc2FjdGlvbnMuCiAgKi8KIGVudW0gbWVtb3J5X3R5 cGUgewogCU1FTU9SWV9ERVZJQ0VfUFJJVkFURSA9IDEsCiAJTUVNT1JZX0RFVklDRV9QVUJMSUMs CiAJTUVNT1JZX0RFVklDRV9GU19EQVgsCisJTUVNT1JZX0RFVklDRV9QQ0lfUDJQRE1BLAogfTsK IAogLyoKZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvbW0uaCBiL2luY2x1ZGUvbGludXgvbW0u aAppbmRleCBhNjFlYmU4YWQ0Y2EuLjIwNTVkZjQxMmE3NyAxMDA2NDQKLS0tIGEvaW5jbHVkZS9s aW51eC9tbS5oCisrKyBiL2luY2x1ZGUvbGludXgvbW0uaApAQCAtODkwLDYgKzg5MCwxOSBAQCBz dGF0aWMgaW5saW5lIGJvb2wgaXNfZGV2aWNlX3B1YmxpY19wYWdlKGNvbnN0IHN0cnVjdCBwYWdl ICpwYWdlKQogCQlwYWdlLT5wZ21hcC0+dHlwZSA9PSBNRU1PUllfREVWSUNFX1BVQkxJQzsKIH0K IAorI2lmZGVmIENPTkZJR19QQ0lfUDJQRE1BCitzdGF0aWMgaW5saW5lIGJvb2wgaXNfcGNpX3Ay cGRtYV9wYWdlKGNvbnN0IHN0cnVjdCBwYWdlICpwYWdlKQoreworCXJldHVybiBpc196b25lX2Rl dmljZV9wYWdlKHBhZ2UpICYmCisJCXBhZ2UtPnBnbWFwLT50eXBlID09IE1FTU9SWV9ERVZJQ0Vf UENJX1AyUERNQTsKK30KKyNlbHNlIC8qIENPTkZJR19QQ0lfUDJQRE1BICovCitzdGF0aWMgaW5s aW5lIGJvb2wgaXNfcGNpX3AycGRtYV9wYWdlKGNvbnN0IHN0cnVjdCBwYWdlICpwYWdlKQorewor CXJldHVybiBmYWxzZTsKK30KKyNlbmRpZiAvKiBDT05GSUdfUENJX1AyUERNQSAqLworCiAjZWxz ZSAvKiBDT05GSUdfREVWX1BBR0VNQVBfT1BTICovCiBzdGF0aWMgaW5saW5lIHZvaWQgZGV2X3Bh Z2VtYXBfZ2V0X29wcyh2b2lkKQogewpAQCAtOTEzLDYgKzkyNiwxMSBAQCBzdGF0aWMgaW5saW5l IGJvb2wgaXNfZGV2aWNlX3B1YmxpY19wYWdlKGNvbnN0IHN0cnVjdCBwYWdlICpwYWdlKQogewog CXJldHVybiBmYWxzZTsKIH0KKworc3RhdGljIGlubGluZSBib29sIGlzX3BjaV9wMnBkbWFfcGFn ZShjb25zdCBzdHJ1Y3QgcGFnZSAqcGFnZSkKK3sKKwlyZXR1cm4gZmFsc2U7Cit9CiAjZW5kaWYg LyogQ09ORklHX0RFVl9QQUdFTUFQX09QUyAqLwogCiBzdGF0aWMgaW5saW5lIHZvaWQgZ2V0X3Bh Z2Uoc3RydWN0IHBhZ2UgKnBhZ2UpCmRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L3BjaS1wMnBk bWEuaCBiL2luY2x1ZGUvbGludXgvcGNpLXAycGRtYS5oCm5ldyBmaWxlIG1vZGUgMTAwNjQ0Cmlu ZGV4IDAwMDAwMDAwMDAwMC4uN2IyYjBmNTQ3NTI4Ci0tLSAvZGV2L251bGwKKysrIGIvaW5jbHVk ZS9saW51eC9wY2ktcDJwZG1hLmgKQEAgLTAsMCArMSwxMDIgQEAKKy8qIFNQRFgtTGljZW5zZS1J ZGVudGlmaWVyOiBHUEwtMi4wICovCisvKgorICogUENJIFBlZXIgMiBQZWVyIERNQSBzdXBwb3J0 LgorICoKKyAqIENvcHlyaWdodCAoYykgMjAxNi0yMDE4LCBMb2dhbiBHdW50aG9ycGUKKyAqIENv cHlyaWdodCAoYykgMjAxNi0yMDE3LCBNaWNyb3NlbWkgQ29ycG9yYXRpb24KKyAqIENvcHlyaWdo dCAoYykgMjAxNywgQ2hyaXN0b3BoIEhlbGx3aWcKKyAqIENvcHlyaWdodCAoYykgMjAxOCwgRWlk ZXRpY29tIEluYy4KKyAqCisgKi8KKworI2lmbmRlZiBfTElOVVhfUENJX1AyUERNQV9ICisjZGVm aW5lIF9MSU5VWF9QQ0lfUDJQRE1BX0gKKworI2luY2x1ZGUgPGxpbnV4L3BjaS5oPgorCitzdHJ1 Y3QgYmxvY2tfZGV2aWNlOworc3RydWN0IHNjYXR0ZXJsaXN0OworCisjaWZkZWYgQ09ORklHX1BD SV9QMlBETUEKK2ludCBwY2lfcDJwZG1hX2FkZF9yZXNvdXJjZShzdHJ1Y3QgcGNpX2RldiAqcGRl diwgaW50IGJhciwgc2l6ZV90IHNpemUsCisJCXU2NCBvZmZzZXQpOworaW50IHBjaV9wMnBkbWFf YWRkX2NsaWVudChzdHJ1Y3QgbGlzdF9oZWFkICpoZWFkLCBzdHJ1Y3QgZGV2aWNlICpkZXYpOwor dm9pZCBwY2lfcDJwZG1hX3JlbW92ZV9jbGllbnQoc3RydWN0IGxpc3RfaGVhZCAqaGVhZCwgc3Ry dWN0IGRldmljZSAqZGV2KTsKK3ZvaWQgcGNpX3AycGRtYV9jbGllbnRfbGlzdF9mcmVlKHN0cnVj dCBsaXN0X2hlYWQgKmhlYWQpOworaW50IHBjaV9wMnBkbWFfZGlzdGFuY2Uoc3RydWN0IHBjaV9k ZXYgKnByb3ZpZGVyLCBzdHJ1Y3QgbGlzdF9oZWFkICpjbGllbnRzLAorCQkJYm9vbCB2ZXJib3Nl KTsKK2Jvb2wgcGNpX3AycGRtYV9hc3NpZ25fcHJvdmlkZXIoc3RydWN0IHBjaV9kZXYgKnByb3Zp ZGVyLAorCQkJCXN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMpOworYm9vbCBwY2lfaGFzX3AycG1l bShzdHJ1Y3QgcGNpX2RldiAqcGRldik7CitzdHJ1Y3QgcGNpX2RldiAqcGNpX3AycG1lbV9maW5k KHN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMpOwordm9pZCAqcGNpX2FsbG9jX3AycG1lbShzdHJ1 Y3QgcGNpX2RldiAqcGRldiwgc2l6ZV90IHNpemUpOwordm9pZCBwY2lfZnJlZV9wMnBtZW0oc3Ry dWN0IHBjaV9kZXYgKnBkZXYsIHZvaWQgKmFkZHIsIHNpemVfdCBzaXplKTsKK3BjaV9idXNfYWRk cl90IHBjaV9wMnBtZW1fdmlydF90b19idXMoc3RydWN0IHBjaV9kZXYgKnBkZXYsIHZvaWQgKmFk ZHIpOworc3RydWN0IHNjYXR0ZXJsaXN0ICpwY2lfcDJwbWVtX2FsbG9jX3NnbChzdHJ1Y3QgcGNp X2RldiAqcGRldiwKKwkJCQkJIHVuc2lnbmVkIGludCAqbmVudHMsIHUzMiBsZW5ndGgpOwordm9p ZCBwY2lfcDJwbWVtX2ZyZWVfc2dsKHN0cnVjdCBwY2lfZGV2ICpwZGV2LCBzdHJ1Y3Qgc2NhdHRl cmxpc3QgKnNnbCk7Cit2b2lkIHBjaV9wMnBtZW1fcHVibGlzaChzdHJ1Y3QgcGNpX2RldiAqcGRl diwgYm9vbCBwdWJsaXNoKTsKKyNlbHNlIC8qIENPTkZJR19QQ0lfUDJQRE1BICovCitzdGF0aWMg aW5saW5lIGludCBwY2lfcDJwZG1hX2FkZF9yZXNvdXJjZShzdHJ1Y3QgcGNpX2RldiAqcGRldiwg aW50IGJhciwKKwkJc2l6ZV90IHNpemUsIHU2NCBvZmZzZXQpCit7CisJcmV0dXJuIC1FT1BOT1RT VVBQOworfQorc3RhdGljIGlubGluZSBpbnQgcGNpX3AycGRtYV9hZGRfY2xpZW50KHN0cnVjdCBs aXN0X2hlYWQgKmhlYWQsCisJCXN0cnVjdCBkZXZpY2UgKmRldikKK3sKKwlyZXR1cm4gMDsKK30K K3N0YXRpYyBpbmxpbmUgdm9pZCBwY2lfcDJwZG1hX3JlbW92ZV9jbGllbnQoc3RydWN0IGxpc3Rf aGVhZCAqaGVhZCwKKwkJc3RydWN0IGRldmljZSAqZGV2KQoreworfQorc3RhdGljIGlubGluZSB2 b2lkIHBjaV9wMnBkbWFfY2xpZW50X2xpc3RfZnJlZShzdHJ1Y3QgbGlzdF9oZWFkICpoZWFkKQor eworfQorc3RhdGljIGlubGluZSBpbnQgcGNpX3AycGRtYV9kaXN0YW5jZShzdHJ1Y3QgcGNpX2Rl diAqcHJvdmlkZXIsCisJCQkJICAgICAgc3RydWN0IGxpc3RfaGVhZCAqY2xpZW50cywKKwkJCQkg ICAgICBib29sIHZlcmJvc2UpCit7CisJcmV0dXJuIC0xOworfQorc3RhdGljIGlubGluZSBib29s IHBjaV9wMnBkbWFfYXNzaWduX3Byb3ZpZGVyKHN0cnVjdCBwY2lfZGV2ICpwcm92aWRlciwKKwkJ CQkJICAgICAgc3RydWN0IGxpc3RfaGVhZCAqY2xpZW50cykKK3sKKwlyZXR1cm4gZmFsc2U7Cit9 CitzdGF0aWMgaW5saW5lIGJvb2wgcGNpX2hhc19wMnBtZW0oc3RydWN0IHBjaV9kZXYgKnBkZXYp Cit7CisJcmV0dXJuIGZhbHNlOworfQorc3RhdGljIGlubGluZSBzdHJ1Y3QgcGNpX2RldiAqcGNp X3AycG1lbV9maW5kKHN0cnVjdCBsaXN0X2hlYWQgKmNsaWVudHMpCit7CisJcmV0dXJuIE5VTEw7 Cit9CitzdGF0aWMgaW5saW5lIHZvaWQgKnBjaV9hbGxvY19wMnBtZW0oc3RydWN0IHBjaV9kZXYg KnBkZXYsIHNpemVfdCBzaXplKQoreworCXJldHVybiBOVUxMOworfQorc3RhdGljIGlubGluZSB2 b2lkIHBjaV9mcmVlX3AycG1lbShzdHJ1Y3QgcGNpX2RldiAqcGRldiwgdm9pZCAqYWRkciwKKwkJ c2l6ZV90IHNpemUpCit7Cit9CitzdGF0aWMgaW5saW5lIHBjaV9idXNfYWRkcl90IHBjaV9wMnBt ZW1fdmlydF90b19idXMoc3RydWN0IHBjaV9kZXYgKnBkZXYsCisJCQkJCQkgICAgdm9pZCAqYWRk cikKK3sKKwlyZXR1cm4gMDsKK30KK3N0YXRpYyBpbmxpbmUgc3RydWN0IHNjYXR0ZXJsaXN0ICpw Y2lfcDJwbWVtX2FsbG9jX3NnbChzdHJ1Y3QgcGNpX2RldiAqcGRldiwKKwkJdW5zaWduZWQgaW50 ICpuZW50cywgdTMyIGxlbmd0aCkKK3sKKwlyZXR1cm4gTlVMTDsKK30KK3N0YXRpYyBpbmxpbmUg dm9pZCBwY2lfcDJwbWVtX2ZyZWVfc2dsKHN0cnVjdCBwY2lfZGV2ICpwZGV2LAorCQlzdHJ1Y3Qg c2NhdHRlcmxpc3QgKnNnbCkKK3sKK30KK3N0YXRpYyBpbmxpbmUgdm9pZCBwY2lfcDJwbWVtX3B1 Ymxpc2goc3RydWN0IHBjaV9kZXYgKnBkZXYsIGJvb2wgcHVibGlzaCkKK3sKK30KKyNlbmRpZiAv KiBDT05GSUdfUENJX1AyUERNQSAqLworI2VuZGlmIC8qIF9MSU5VWF9QQ0lfUDJQX0ggKi8KZGlm ZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvcGNpLmggYi9pbmNsdWRlL2xpbnV4L3BjaS5oCmluZGV4 IGU3MmNhOGRkNjI0MS4uNWQ5NWRiZjIxZjRhIDEwMDY0NAotLS0gYS9pbmNsdWRlL2xpbnV4L3Bj aS5oCisrKyBiL2luY2x1ZGUvbGludXgvcGNpLmgKQEAgLTI4MSw2ICsyODEsNyBAQCBzdHJ1Y3Qg cGNpZV9saW5rX3N0YXRlOwogc3RydWN0IHBjaV92cGQ7CiBzdHJ1Y3QgcGNpX3NyaW92Owogc3Ry dWN0IHBjaV9hdHM7CitzdHJ1Y3QgcGNpX3AycGRtYTsKIAogLyogVGhlIHBjaV9kZXYgc3RydWN0 dXJlIGRlc2NyaWJlcyBQQ0kgZGV2aWNlcyAqLwogc3RydWN0IHBjaV9kZXYgewpAQCAtNDM4LDYg KzQzOSw5IEBAIHN0cnVjdCBwY2lfZGV2IHsKICNlbmRpZgogI2lmZGVmIENPTkZJR19QQ0lfUEFT SUQKIAl1MTYJCXBhc2lkX2ZlYXR1cmVzOworI2VuZGlmCisjaWZkZWYgQ09ORklHX1BDSV9QMlBE TUEKKwlzdHJ1Y3QgcGNpX3AycGRtYSAqcDJwZG1hOwogI2VuZGlmCiAJcGh5c19hZGRyX3QJcm9t OwkJLyogUGh5c2ljYWwgYWRkcmVzcyBpZiBub3QgZnJvbSBCQVIgKi8KIAlzaXplX3QJCXJvbWxl bjsJCS8qIExlbmd0aCBpZiBub3QgZnJvbSBCQVIgKi8KLS0gCjIuMTkuMAoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTGludXgtbnZkaW1tIG1haWxpbmcg bGlzdApMaW51eC1udmRpbW1AbGlzdHMuMDEub3JnCmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxt YW4vbGlzdGluZm8vbGludXgtbnZkaW1tCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: logang@deltatee.com (Logan Gunthorpe) Date: Wed, 12 Sep 2018 18:11:44 -0600 Subject: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory In-Reply-To: <20180913001156.4115-1-logang@deltatee.com> References: <20180913001156.4115-1-logang@deltatee.com> Message-ID: <20180913001156.4115-2-logang@deltatee.com> Some PCI devices may have memory mapped in a BAR space that's intended for use in peer-to-peer transactions. In order to enable such transactions the memory must be registered with ZONE_DEVICE pages so it can be used by DMA interfaces in existing drivers. Add an interface for other subsystems to find and allocate chunks of P2P memory as necessary to facilitate transfers between two PCI peers: int pci_p2pdma_add_client(); struct pci_dev *pci_p2pmem_find(); void *pci_alloc_p2pmem(); The new interface requires a driver to collect a list of client devices involved in the transaction with the pci_p2pmem_add_client*() functions then call pci_p2pmem_find() to obtain any suitable P2P memory. Once this is done the list is bound to the memory and the calling driver is free to add and remove clients as necessary (adding incompatible clients will fail). With a suitable p2pmem device, memory can then be allocated with pci_alloc_p2pmem() for use in DMA transactions. Depending on hardware, using peer-to-peer memory may reduce the bandwidth of the transfer but can significantly reduce pressure on system memory. This may be desirable in many cases: for example a system could be designed with a small CPU connected to a PCIe switch by a small number of lanes which would maximize the number of lanes available to connect to NVMe devices. The code is designed to only utilize the p2pmem device if all the devices involved in a transfer are behind the same PCI bridge. This is because we have no way of knowing whether peer-to-peer routing between PCIe Root Ports is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P transfers that go through the RC is limited to only reducing DRAM usage and, in some cases, coding convenience. The PCI-SIG may be exploring adding a new capability bit to advertise whether this is possible for future hardware. This commit includes significant rework and feedback from Christoph Hellwig. Signed-off-by: Christoph Hellwig Signed-off-by: Logan Gunthorpe --- drivers/pci/Kconfig | 17 + drivers/pci/Makefile | 1 + drivers/pci/p2pdma.c | 761 +++++++++++++++++++++++++++++++++++++ include/linux/memremap.h | 5 + include/linux/mm.h | 18 + include/linux/pci-p2pdma.h | 102 +++++ include/linux/pci.h | 4 + 7 files changed, 908 insertions(+) create mode 100644 drivers/pci/p2pdma.c create mode 100644 include/linux/pci-p2pdma.h diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 56ff8f6d31fc..deb68be4fdac 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -132,6 +132,23 @@ config PCI_PASID If unsure, say N. +config PCI_P2PDMA + bool "PCI peer-to-peer transfer support" + depends on PCI && ZONE_DEVICE + select GENERIC_ALLOCATOR + help + Enable? drivers to do PCI peer-to-peer transactions to and from + BARs that are exposed in other devices that are the part of + the hierarchy where peer-to-peer DMA is guaranteed by the PCI + specification to work (ie. anything below a single PCI bridge). + + Many PCIe root complexes do not support P2P transactions and + it's hard to tell which support it at all, so at this time, + P2P DMA transations must be between devices behind the same root + port. + + If unsure, say N. + config PCI_LABEL def_bool y if (DMI || ACPI) depends on PCI diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile index 1b2cfe51e8d7..85f4a703b2be 100644 --- a/drivers/pci/Makefile +++ b/drivers/pci/Makefile @@ -26,6 +26,7 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o obj-$(CONFIG_PCI_STUB) += pci-stub.o obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o obj-$(CONFIG_PCI_ECAM) += ecam.o +obj-$(CONFIG_PCI_P2PDMA) += p2pdma.o obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o # Endpoint library must be initialized before its users diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c new file mode 100644 index 000000000000..88aaec5351cd --- /dev/null +++ b/drivers/pci/p2pdma.c @@ -0,0 +1,761 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * PCI Peer 2 Peer DMA support. + * + * Copyright (c) 2016-2018, Logan Gunthorpe + * Copyright (c) 2016-2017, Microsemi Corporation + * Copyright (c) 2017, Christoph Hellwig + * Copyright (c) 2018, Eideticom Inc. + */ + +#define pr_fmt(fmt) "pci-p2pdma: " fmt +#include +#include +#include +#include +#include +#include +#include +#include + +struct pci_p2pdma { + struct percpu_ref devmap_ref; + struct completion devmap_ref_done; + struct gen_pool *pool; + bool p2pmem_published; +}; + +static void pci_p2pdma_percpu_release(struct percpu_ref *ref) +{ + struct pci_p2pdma *p2p = + container_of(ref, struct pci_p2pdma, devmap_ref); + + complete_all(&p2p->devmap_ref_done); +} + +static void pci_p2pdma_percpu_kill(void *data) +{ + struct percpu_ref *ref = data; + + if (percpu_ref_is_dying(ref)) + return; + + percpu_ref_kill(ref); +} + +static void pci_p2pdma_release(void *data) +{ + struct pci_dev *pdev = data; + + if (!pdev->p2pdma) + return; + + wait_for_completion(&pdev->p2pdma->devmap_ref_done); + percpu_ref_exit(&pdev->p2pdma->devmap_ref); + + gen_pool_destroy(pdev->p2pdma->pool); + pdev->p2pdma = NULL; +} + +static int pci_p2pdma_setup(struct pci_dev *pdev) +{ + int error = -ENOMEM; + struct pci_p2pdma *p2p; + + p2p = devm_kzalloc(&pdev->dev, sizeof(*p2p), GFP_KERNEL); + if (!p2p) + return -ENOMEM; + + p2p->pool = gen_pool_create(PAGE_SHIFT, dev_to_node(&pdev->dev)); + if (!p2p->pool) + goto out; + + init_completion(&p2p->devmap_ref_done); + error = percpu_ref_init(&p2p->devmap_ref, + pci_p2pdma_percpu_release, 0, GFP_KERNEL); + if (error) + goto out_pool_destroy; + + percpu_ref_switch_to_atomic_sync(&p2p->devmap_ref); + + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev); + if (error) + goto out_pool_destroy; + + pdev->p2pdma = p2p; + + return 0; + +out_pool_destroy: + gen_pool_destroy(p2p->pool); +out: + devm_kfree(&pdev->dev, p2p); + return error; +} + +/** + * pci_p2pdma_add_resource - add memory for use as p2p memory + * @pdev: the device to add the memory to + * @bar: PCI BAR to add + * @size: size of the memory to add, may be zero to use the whole BAR + * @offset: offset into the PCI BAR + * + * The memory will be given ZONE_DEVICE struct pages so that it may + * be used with any DMA request. + */ +int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, + u64 offset) +{ + struct dev_pagemap *pgmap; + void *addr; + int error; + + if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM)) + return -EINVAL; + + if (offset >= pci_resource_len(pdev, bar)) + return -EINVAL; + + if (!size) + size = pci_resource_len(pdev, bar) - offset; + + if (size + offset > pci_resource_len(pdev, bar)) + return -EINVAL; + + if (!pdev->p2pdma) { + error = pci_p2pdma_setup(pdev); + if (error) + return error; + } + + pgmap = devm_kzalloc(&pdev->dev, sizeof(*pgmap), GFP_KERNEL); + if (!pgmap) + return -ENOMEM; + + pgmap->res.start = pci_resource_start(pdev, bar) + offset; + pgmap->res.end = pgmap->res.start + size - 1; + pgmap->res.flags = pci_resource_flags(pdev, bar); + pgmap->ref = &pdev->p2pdma->devmap_ref; + pgmap->type = MEMORY_DEVICE_PCI_P2PDMA; + + addr = devm_memremap_pages(&pdev->dev, pgmap); + if (IS_ERR(addr)) { + error = PTR_ERR(addr); + goto pgmap_free; + } + + error = gen_pool_add_virt(pdev->p2pdma->pool, (unsigned long)addr, + pci_bus_address(pdev, bar) + offset, + resource_size(&pgmap->res), dev_to_node(&pdev->dev)); + if (error) + goto pgmap_free; + + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_percpu_kill, + &pdev->p2pdma->devmap_ref); + if (error) + goto pgmap_free; + + pci_info(pdev, "added peer-to-peer DMA memory %pR\n", + &pgmap->res); + + return 0; + +pgmap_free: + devres_free(pgmap); + return error; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_add_resource); + +static struct pci_dev *find_parent_pci_dev(struct device *dev) +{ + struct device *parent; + + dev = get_device(dev); + + while (dev) { + if (dev_is_pci(dev)) + return to_pci_dev(dev); + + parent = get_device(dev->parent); + put_device(dev); + dev = parent; + } + + return NULL; +} + +/* + * Check if a PCI bridge has it's ACS redirection bits set to redirect P2P + * TLPs upstream via ACS. Returns 1 if the packets will be redirected + * upstream, 0 otherwise. + */ +static int pci_bridge_has_acs_redir(struct pci_dev *dev) +{ + int pos; + u16 ctrl; + + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS); + if (!pos) + return 0; + + pci_read_config_word(dev, pos + PCI_ACS_CTRL, &ctrl); + + if (ctrl & (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_EC)) + return 1; + + return 0; +} + +static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *dev) +{ + if (!buf) + return; + + seq_buf_printf(buf, "%04x:%02x:%02x.%x;", pci_domain_nr(dev->bus), + dev->bus->number, PCI_SLOT(dev->devfn), + PCI_FUNC(dev->devfn)); +} + +/* + * Find the distance through the nearest common upstream bridge between + * two PCI devices. + * + * If the two devices are the same device then 0 will be returned. + * + * If there are two virtual functions of the same device behind the same + * bridge port then 2 will be returned (one step down to the PCIe switch, + * then one step back to the same device). + * + * In the case where two devices are connected to the same PCIe switch, the + * value 4 will be returned. This corresponds to the following PCI tree: + * + * -+ Root Port + * \+ Switch Upstream Port + * +-+ Switch Downstream Port + * + \- Device A + * \-+ Switch Downstream Port + * \- Device B + * + * The distance is 4 because we traverse from Device A through the downstream + * port of the switch, to the common upstream port, back up to the second + * downstream port and then to Device B. + * + * Any two devices that don't have a common upstream bridge will return -1. + * In this way devices on separate PCIe root ports will be rejected, which + * is what we want for peer-to-peer seeing each PCIe root port defines a + * separate hierarchy domain and there's no way to determine whether the root + * complex supports forwarding between them. + * + * In the case where two devices are connected to different PCIe switches, + * this function will still return a positive distance as long as both + * switches evenutally have a common upstream bridge. Note this covers + * the case of using multiple PCIe switches to achieve a desired level of + * fan-out from a root port. The exact distance will be a function of the + * number of switches between Device A and Device B. + * + * If a bridge which has any ACS redirection bits set is in the path + * then this functions will return -2. This is so we reject any + * cases where the TLPs are forwarded up into the root complex. + * In this case, a list of all infringing bridge addresses will be + * populated in acs_list (assuming it's non-null) for printk purposes. + */ +static int upstream_bridge_distance(struct pci_dev *a, + struct pci_dev *b, + struct seq_buf *acs_list) +{ + int dist_a = 0; + int dist_b = 0; + struct pci_dev *bb = NULL; + int acs_cnt = 0; + + /* + * Note, we don't need to take references to devices returned by + * pci_upstream_bridge() seeing we hold a reference to a child + * device which will already hold a reference to the upstream bridge. + */ + + while (a) { + dist_b = 0; + + if (pci_bridge_has_acs_redir(a)) { + seq_buf_print_bus_devfn(acs_list, a); + acs_cnt++; + } + + bb = b; + + while (bb) { + if (a == bb) + goto check_b_path_acs; + + bb = pci_upstream_bridge(bb); + dist_b++; + } + + a = pci_upstream_bridge(a); + dist_a++; + } + + return -1; + +check_b_path_acs: + bb = b; + + while (bb) { + if (a == bb) + break; + + if (pci_bridge_has_acs_redir(bb)) { + seq_buf_print_bus_devfn(acs_list, bb); + acs_cnt++; + } + + bb = pci_upstream_bridge(bb); + } + + if (acs_cnt) + return -2; + + return dist_a + dist_b; +} + +static int upstream_bridge_distance_warn(struct pci_dev *provider, + struct pci_dev *client) +{ + struct seq_buf acs_list; + int ret; + + seq_buf_init(&acs_list, kmalloc(PAGE_SIZE, GFP_KERNEL), PAGE_SIZE); + + ret = upstream_bridge_distance(provider, client, &acs_list); + if (ret == -2) { + pci_warn(client, "cannot be used for peer-to-peer DMA as ACS redirect is set between the client and provider\n"); + /* Drop final semicolon */ + acs_list.buffer[acs_list.len-1] = 0; + pci_warn(client, "to disable ACS redirect for this path, add the kernel parameter: pci=disable_acs_redir=%s\n", + acs_list.buffer); + + } else if (ret < 0) { + pci_warn(client, "cannot be used for peer-to-peer DMA as the client and provider do not share an upstream bridge\n"); + } + + kfree(acs_list.buffer); + + return ret; +} + +struct pci_p2pdma_client { + struct list_head list; + struct pci_dev *client; + struct pci_dev *provider; +}; + +/** + * pci_p2pdma_add_client - allocate a new element in a client device list + * @head: list head of p2pdma clients + * @dev: device to add to the list + * + * This adds @dev to a list of clients used by a p2pdma device. + * This list should be passed to pci_p2pmem_find(). Once pci_p2pmem_find() has + * been called successfully, the list will be bound to a specific p2pdma + * device and new clients can only be added to the list if they are + * supported by that p2pdma device. + * + * The caller is expected to have a lock which protects @head as necessary + * so that none of the pci_p2p functions can be called concurrently + * on that list. + * + * Returns 0 if the client was successfully added. + */ +int pci_p2pdma_add_client(struct list_head *head, struct device *dev) +{ + struct pci_p2pdma_client *item, *new_item; + struct pci_dev *provider = NULL; + struct pci_dev *client; + int ret; + + if (IS_ENABLED(CONFIG_DMA_VIRT_OPS) && dev->dma_ops == &dma_virt_ops) { + dev_warn(dev, "cannot be used for peer-to-peer DMA because the driver makes use of dma_virt_ops\n"); + return -ENODEV; + } + + client = find_parent_pci_dev(dev); + if (!client) { + dev_warn(dev, "cannot be used for peer-to-peer DMA as it is not a PCI device\n"); + return -ENODEV; + } + + item = list_first_entry_or_null(head, struct pci_p2pdma_client, list); + if (item && item->provider) { + provider = item->provider; + + ret = upstream_bridge_distance_warn(provider, client); + if (ret < 0) { + ret = -EXDEV; + goto put_client; + } + } + + new_item = kzalloc(sizeof(*new_item), GFP_KERNEL); + if (!new_item) { + ret = -ENOMEM; + goto put_client; + } + + new_item->client = client; + new_item->provider = pci_dev_get(provider); + + list_add_tail(&new_item->list, head); + + return 0; + +put_client: + pci_dev_put(client); + return ret; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_add_client); + +static void pci_p2pdma_client_free(struct pci_p2pdma_client *item) +{ + list_del(&item->list); + pci_dev_put(item->client); + pci_dev_put(item->provider); + kfree(item); +} + +/** + * pci_p2pdma_remove_client - remove and free a p2pdma client + * @head: list head of p2pdma clients + * @dev: device to remove from the list + * + * This removes @dev from a list of clients used by a p2pdma device. + * The caller is expected to have a lock which protects @head as necessary + * so that none of the pci_p2p functions can be called concurrently + * on that list. + */ +void pci_p2pdma_remove_client(struct list_head *head, struct device *dev) +{ + struct pci_p2pdma_client *pos, *tmp; + struct pci_dev *pdev; + + pdev = find_parent_pci_dev(dev); + if (!pdev) + return; + + list_for_each_entry_safe(pos, tmp, head, list) { + if (pos->client != pdev) + continue; + + pci_p2pdma_client_free(pos); + } + + pci_dev_put(pdev); +} +EXPORT_SYMBOL_GPL(pci_p2pdma_remove_client); + +/** + * pci_p2pdma_client_list_free - free an entire list of p2pdma clients + * @head: list head of p2pdma clients + * + * This removes all devices in a list of clients used by a p2pdma device. + * The caller is expected to have a lock which protects @head as necessary + * so that none of the pci_p2pdma functions can be called concurrently + * on that list. + */ +void pci_p2pdma_client_list_free(struct list_head *head) +{ + struct pci_p2pdma_client *pos, *tmp; + + list_for_each_entry_safe(pos, tmp, head, list) + pci_p2pdma_client_free(pos); +} +EXPORT_SYMBOL_GPL(pci_p2pdma_client_list_free); + +/** + * pci_p2pdma_distance - Determive the cumulative distance between + * a p2pdma provider and the clients in use. + * @provider: p2pdma provider to check against the client list + * @clients: list of devices to check (NULL-terminated) + * @verbose: if true, print warnings for devices when we return -1 + * + * Returns -1 if any of the clients are not compatible (behind the same + * root port as the provider), otherwise returns a positive number where + * the lower number is the preferrable choice. (If there's one client + * that's the same as the provider it will return 0, which is best choice). + * + * For now, "compatible" means the provider and the clients are all behind + * the same PCI root port. This cuts out cases that may work but is safest + * for the user. Future work can expand this to white-list root complexes that + * can safely forward between each ports. + */ +int pci_p2pdma_distance(struct pci_dev *provider, struct list_head *clients, + bool verbose) +{ + struct pci_p2pdma_client *pos; + int ret; + int distance = 0; + bool not_supported = false; + + if (list_empty(clients)) + return -1; + + list_for_each_entry(pos, clients, list) { + if (verbose) + ret = upstream_bridge_distance_warn(provider, + pos->client); + else + ret = upstream_bridge_distance(provider, pos->client, + NULL); + + if (ret < 0) + not_supported = true; + + if (not_supported && !verbose) + break; + + distance += ret; + } + + if (not_supported) + return -1; + + return distance; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_distance); + +/** + * pci_p2pdma_assign_provider - Check compatibily (as per pci_p2pdma_distance) + * and assign a provider to a list of clients + * @provider: p2pdma provider to assign to the client list + * @clients: list of devices to check (NULL-terminated) + * + * Returns false if any of the clients are not compatible, true if the + * provider was successfully assigned to the clients. + */ +bool pci_p2pdma_assign_provider(struct pci_dev *provider, + struct list_head *clients) +{ + struct pci_p2pdma_client *pos; + + if (pci_p2pdma_distance(provider, clients, true) < 0) + return false; + + list_for_each_entry(pos, clients, list) + pos->provider = provider; + + return true; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_assign_provider); + +/** + * pci_has_p2pmem - check if a given PCI device has published any p2pmem + * @pdev: PCI device to check + */ +bool pci_has_p2pmem(struct pci_dev *pdev) +{ + return pdev->p2pdma && pdev->p2pdma->p2pmem_published; +} +EXPORT_SYMBOL_GPL(pci_has_p2pmem); + +/** + * pci_p2pmem_find - find a peer-to-peer DMA memory device compatible with + * the specified list of clients and shortest distance (as determined + * by pci_p2pmem_dma()) + * @clients: list of devices to check (NULL-terminated) + * + * If multiple devices are behind the same switch, the one "closest" to the + * client devices in use will be chosen first. (So if one of the providers are + * the same as one of the clients, that provider will be used ahead of any + * other providers that are unrelated). If multiple providers are an equal + * distance away, one will be chosen at random. + * + * Returns a pointer to the PCI device with a reference taken (use pci_dev_put + * to return the reference) or NULL if no compatible device is found. The + * found provider will also be assigned to the client list. + */ +struct pci_dev *pci_p2pmem_find(struct list_head *clients) +{ + struct pci_dev *pdev = NULL; + struct pci_p2pdma_client *pos; + int distance; + int closest_distance = INT_MAX; + struct pci_dev **closest_pdevs; + int dev_cnt = 0; + const int max_devs = PAGE_SIZE / sizeof(*closest_pdevs); + int i; + + closest_pdevs = kmalloc(PAGE_SIZE, GFP_KERNEL); + + while ((pdev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, pdev))) { + if (!pci_has_p2pmem(pdev)) + continue; + + distance = pci_p2pdma_distance(pdev, clients, false); + if (distance < 0 || distance > closest_distance) + continue; + + if (distance == closest_distance && dev_cnt >= max_devs) + continue; + + if (distance < closest_distance) { + for (i = 0; i < dev_cnt; i++) + pci_dev_put(closest_pdevs[i]); + + dev_cnt = 0; + closest_distance = distance; + } + + closest_pdevs[dev_cnt++] = pci_dev_get(pdev); + } + + if (dev_cnt) + pdev = pci_dev_get(closest_pdevs[prandom_u32_max(dev_cnt)]); + + for (i = 0; i < dev_cnt; i++) + pci_dev_put(closest_pdevs[i]); + + if (pdev) + list_for_each_entry(pos, clients, list) + pos->provider = pdev; + + kfree(closest_pdevs); + return pdev; +} +EXPORT_SYMBOL_GPL(pci_p2pmem_find); + +/** + * pci_alloc_p2p_mem - allocate peer-to-peer DMA memory + * @pdev: the device to allocate memory from + * @size: number of bytes to allocate + * + * Returns the allocated memory or NULL on error. + */ +void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size) +{ + void *ret; + + if (unlikely(!pdev->p2pdma)) + return NULL; + + if (unlikely(!percpu_ref_tryget_live(&pdev->p2pdma->devmap_ref))) + return NULL; + + ret = (void *)gen_pool_alloc(pdev->p2pdma->pool, size); + + if (unlikely(!ret)) + percpu_ref_put(&pdev->p2pdma->devmap_ref); + + return ret; +} +EXPORT_SYMBOL_GPL(pci_alloc_p2pmem); + +/** + * pci_free_p2pmem - allocate peer-to-peer DMA memory + * @pdev: the device the memory was allocated from + * @addr: address of the memory that was allocated + * @size: number of bytes that was allocated + */ +void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size) +{ + gen_pool_free(pdev->p2pdma->pool, (uintptr_t)addr, size); + percpu_ref_put(&pdev->p2pdma->devmap_ref); +} +EXPORT_SYMBOL_GPL(pci_free_p2pmem); + +/** + * pci_virt_to_bus - return the PCI bus address for a given virtual + * address obtained with pci_alloc_p2pmem() + * @pdev: the device the memory was allocated from + * @addr: address of the memory that was allocated + */ +pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr) +{ + if (!addr) + return 0; + if (!pdev->p2pdma) + return 0; + + /* + * Note: when we added the memory to the pool we used the PCI + * bus address as the physical address. So gen_pool_virt_to_phys() + * actually returns the bus address despite the misleading name. + */ + return gen_pool_virt_to_phys(pdev->p2pdma->pool, (unsigned long)addr); +} +EXPORT_SYMBOL_GPL(pci_p2pmem_virt_to_bus); + +/** + * pci_p2pmem_alloc_sgl - allocate peer-to-peer DMA memory in a scatterlist + * @pdev: the device to allocate memory from + * @sgl: the allocated scatterlist + * @nents: the number of SG entries in the list + * @length: number of bytes to allocate + * + * Returns 0 on success + */ +struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, + unsigned int *nents, u32 length) +{ + struct scatterlist *sg; + void *addr; + + sg = kzalloc(sizeof(*sg), GFP_KERNEL); + if (!sg) + return NULL; + + sg_init_table(sg, 1); + + addr = pci_alloc_p2pmem(pdev, length); + if (!addr) + goto out_free_sg; + + sg_set_buf(sg, addr, length); + *nents = 1; + return sg; + +out_free_sg: + kfree(sg); + return NULL; +} +EXPORT_SYMBOL_GPL(pci_p2pmem_alloc_sgl); + +/** + * pci_p2pmem_free_sgl - free a scatterlist allocated by pci_p2pmem_alloc_sgl() + * @pdev: the device to allocate memory from + * @sgl: the allocated scatterlist + * @nents: the number of SG entries in the list + */ +void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl) +{ + struct scatterlist *sg; + int count; + + for_each_sg(sgl, sg, INT_MAX, count) { + if (!sg) + break; + + pci_free_p2pmem(pdev, sg_virt(sg), sg->length); + } + kfree(sgl); +} +EXPORT_SYMBOL_GPL(pci_p2pmem_free_sgl); + +/** + * pci_p2pmem_publish - publish the peer-to-peer DMA memory for use by + * other devices with pci_p2pmem_find() + * @pdev: the device with peer-to-peer DMA memory to publish + * @publish: set to true to publish the memory, false to unpublish it + * + * Published memory can be used by other PCI device drivers for + * peer-2-peer DMA operations. Non-published memory is reserved for + * exlusive use of the device driver that registers the peer-to-peer + * memory. + */ +void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) +{ + if (publish && !pdev->p2pdma) + return; + + pdev->p2pdma->p2pmem_published = publish; +} +EXPORT_SYMBOL_GPL(pci_p2pmem_publish); diff --git a/include/linux/memremap.h b/include/linux/memremap.h index f91f9e763557..9553370ebdad 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -53,11 +53,16 @@ struct vmem_altmap { * wakeup event whenever a page is unpinned and becomes idle. This * wakeup is used to coordinate physical address space management (ex: * fs truncate/hole punch) vs pinned pages (ex: device dma). + * + * MEMORY_DEVICE_PCI_P2PDMA: + * Device memory residing in a PCI BAR intended for use with Peer-to-Peer + * transactions. */ enum memory_type { MEMORY_DEVICE_PRIVATE = 1, MEMORY_DEVICE_PUBLIC, MEMORY_DEVICE_FS_DAX, + MEMORY_DEVICE_PCI_P2PDMA, }; /* diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..2055df412a77 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -890,6 +890,19 @@ static inline bool is_device_public_page(const struct page *page) page->pgmap->type == MEMORY_DEVICE_PUBLIC; } +#ifdef CONFIG_PCI_P2PDMA +static inline bool is_pci_p2pdma_page(const struct page *page) +{ + return is_zone_device_page(page) && + page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA; +} +#else /* CONFIG_PCI_P2PDMA */ +static inline bool is_pci_p2pdma_page(const struct page *page) +{ + return false; +} +#endif /* CONFIG_PCI_P2PDMA */ + #else /* CONFIG_DEV_PAGEMAP_OPS */ static inline void dev_pagemap_get_ops(void) { @@ -913,6 +926,11 @@ static inline bool is_device_public_page(const struct page *page) { return false; } + +static inline bool is_pci_p2pdma_page(const struct page *page) +{ + return false; +} #endif /* CONFIG_DEV_PAGEMAP_OPS */ static inline void get_page(struct page *page) diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h new file mode 100644 index 000000000000..7b2b0f547528 --- /dev/null +++ b/include/linux/pci-p2pdma.h @@ -0,0 +1,102 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * PCI Peer 2 Peer DMA support. + * + * Copyright (c) 2016-2018, Logan Gunthorpe + * Copyright (c) 2016-2017, Microsemi Corporation + * Copyright (c) 2017, Christoph Hellwig + * Copyright (c) 2018, Eideticom Inc. + * + */ + +#ifndef _LINUX_PCI_P2PDMA_H +#define _LINUX_PCI_P2PDMA_H + +#include + +struct block_device; +struct scatterlist; + +#ifdef CONFIG_PCI_P2PDMA +int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, + u64 offset); +int pci_p2pdma_add_client(struct list_head *head, struct device *dev); +void pci_p2pdma_remove_client(struct list_head *head, struct device *dev); +void pci_p2pdma_client_list_free(struct list_head *head); +int pci_p2pdma_distance(struct pci_dev *provider, struct list_head *clients, + bool verbose); +bool pci_p2pdma_assign_provider(struct pci_dev *provider, + struct list_head *clients); +bool pci_has_p2pmem(struct pci_dev *pdev); +struct pci_dev *pci_p2pmem_find(struct list_head *clients); +void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size); +void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size); +pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr); +struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, + unsigned int *nents, u32 length); +void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl); +void pci_p2pmem_publish(struct pci_dev *pdev, bool publish); +#else /* CONFIG_PCI_P2PDMA */ +static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, + size_t size, u64 offset) +{ + return -EOPNOTSUPP; +} +static inline int pci_p2pdma_add_client(struct list_head *head, + struct device *dev) +{ + return 0; +} +static inline void pci_p2pdma_remove_client(struct list_head *head, + struct device *dev) +{ +} +static inline void pci_p2pdma_client_list_free(struct list_head *head) +{ +} +static inline int pci_p2pdma_distance(struct pci_dev *provider, + struct list_head *clients, + bool verbose) +{ + return -1; +} +static inline bool pci_p2pdma_assign_provider(struct pci_dev *provider, + struct list_head *clients) +{ + return false; +} +static inline bool pci_has_p2pmem(struct pci_dev *pdev) +{ + return false; +} +static inline struct pci_dev *pci_p2pmem_find(struct list_head *clients) +{ + return NULL; +} +static inline void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size) +{ + return NULL; +} +static inline void pci_free_p2pmem(struct pci_dev *pdev, void *addr, + size_t size) +{ +} +static inline pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, + void *addr) +{ + return 0; +} +static inline struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, + unsigned int *nents, u32 length) +{ + return NULL; +} +static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev, + struct scatterlist *sgl) +{ +} +static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) +{ +} +#endif /* CONFIG_PCI_P2PDMA */ +#endif /* _LINUX_PCI_P2P_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index e72ca8dd6241..5d95dbf21f4a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -281,6 +281,7 @@ struct pcie_link_state; struct pci_vpd; struct pci_sriov; struct pci_ats; +struct pci_p2pdma; /* The pci_dev structure describes PCI devices */ struct pci_dev { @@ -438,6 +439,9 @@ struct pci_dev { #endif #ifdef CONFIG_PCI_PASID u16 pasid_features; +#endif +#ifdef CONFIG_PCI_P2PDMA + struct pci_p2pdma *p2pdma; #endif phys_addr_t rom; /* Physical address if not from BAR */ size_t romlen; /* Length if not from BAR */ -- 2.19.0