From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com (ale.deltatee.com [207.54.116.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id EDC9721107824 for ; Thu, 30 Aug 2018 11:54:07 -0700 (PDT) From: Logan Gunthorpe Date: Thu, 30 Aug 2018 12:53:52 -0600 Message-Id: <20180830185352.3369-14-logang@deltatee.com> In-Reply-To: <20180830185352.3369-1-logang@deltatee.com> References: <20180830185352.3369-1-logang@deltatee.com> MIME-Version: 1.0 Subject: [PATCH v5 13/13] nvmet: Optionally use PCI P2P memory List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: =?UTF-8?q?Christian=20K=C3=B6nig?= , Benjamin Herrenschmidt , Steve Wise , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: V2UgY3JlYXRlIGEgY29uZmlnZnMgYXR0cmlidXRlIGluIGVhY2ggbnZtZS1mYWJyaWNzIHRhcmdl dCBwb3J0IHRvCmVuYWJsZSBwMnAgbWVtb3J5IHVzZS4gV2hlbiBlbmFibGVkLCB0aGUgcG9ydCB3 aWxsIG9ubHkgdGhlbiB1c2UgdGhlCnAycCBtZW1vcnkgaWYgYSBwMnAgbWVtb3J5IGRldmljZSBj YW4gYmUgZm91bmQgd2hpY2ggaXMgYmVoaW5kIHRoZQpzYW1lIHN3aXRjaCBoaWVyYXJjaHkgYXMg dGhlIFJETUEgcG9ydCBhbmQgYWxsIHRoZSBibG9jayBkZXZpY2VzIGluCnVzZS4gSWYgdGhlIHVz ZXIgZW5hYmxlZCBpdCBhbmQgbm8gZGV2aWNlcyBhcmUgZm91bmQsIHRoZW4gdGhlIHN5c3RlbQp3 aWxsIHNpbGVudGx5IGZhbGwgYmFjayBvbiB1c2luZyByZWd1bGFyIG1lbW9yeS4KCklmIGFwcHJv cHJpYXRlLCB0aGF0IHBvcnQgd2lsbCBhbGxvY2F0ZSBtZW1vcnkgZm9yIHRoZSBSRE1BIGJ1ZmZl cnMKZm9yIHF1ZXVlcyBmcm9tIHRoZSBwMnBtZW0gZGV2aWNlIGZhbGxpbmcgYmFjayB0byBzeXN0 ZW0gbWVtb3J5IHNob3VsZAphbnl0aGluZyBmYWlsLgoKSWRlYWxseSwgd2UnZCB3YW50IHRvIHVz ZSBhbiBOVk1FIENNQiBidWZmZXIgYXMgcDJwIG1lbW9yeS4gVGhpcyB3b3VsZApzYXZlIGFuIGV4 dHJhIFBDSSB0cmFuc2ZlciBhcyB0aGUgTlZNRSBjYXJkIGNvdWxkIGp1c3QgdGFrZSB0aGUgZGF0 YQpvdXQgb2YgaXQncyBvd24gbWVtb3J5LiBIb3dldmVyLCBhdCB0aGlzIHRpbWUsIG9ubHkgYSBs aW1pdGVkIG51bWJlcgpvZiBjYXJkcyB3aXRoIENNQiBidWZmZXJzIHNlZW0gdG8gYmUgYXZhaWxh YmxlLgoKU2lnbmVkLW9mZi1ieTogU3RlcGhlbiBCYXRlcyA8c2JhdGVzQHJhaXRobGluLmNvbT4K U2lnbmVkLW9mZi1ieTogU3RldmUgV2lzZSA8c3dpc2VAb3BlbmdyaWRjb21wdXRpbmcuY29tPgpb aGNoOiBwYXJ0aWFsIHJld3JpdGUgb2YgdGhlIGluaXRpYWwgY29kZV0KU2lnbmVkLW9mZi1ieTog Q2hyaXN0b3BoIEhlbGx3aWcgPGhjaEBsc3QuZGU+ClNpZ25lZC1vZmYtYnk6IExvZ2FuIEd1bnRo b3JwZSA8bG9nYW5nQGRlbHRhdGVlLmNvbT4KLS0tCiBkcml2ZXJzL252bWUvdGFyZ2V0L2NvbmZp Z2ZzLmMgfCAgMzYgKysrKysrKysrKysKIGRyaXZlcnMvbnZtZS90YXJnZXQvY29yZS5jICAgICB8 IDEzMyArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrLQogZHJpdmVycy9u dm1lL3RhcmdldC9udm1ldC5oICAgIHwgIDEzICsrKysKIGRyaXZlcnMvbnZtZS90YXJnZXQvcmRt YS5jICAgICB8ICAgMiArCiA0IGZpbGVzIGNoYW5nZWQsIDE4MyBpbnNlcnRpb25zKCspLCAxIGRl bGV0aW9uKC0pCgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9udm1lL3RhcmdldC9jb25maWdmcy5jIGIv ZHJpdmVycy9udm1lL3RhcmdldC9jb25maWdmcy5jCmluZGV4IGIzN2E4ZTNlM2Y4MC4uMGRmYjBl MGMzZDIxIDEwMDY0NAotLS0gYS9kcml2ZXJzL252bWUvdGFyZ2V0L2NvbmZpZ2ZzLmMKKysrIGIv ZHJpdmVycy9udm1lL3RhcmdldC9jb25maWdmcy5jCkBAIC0xNyw2ICsxNyw4IEBACiAjaW5jbHVk ZSA8bGludXgvc2xhYi5oPgogI2luY2x1ZGUgPGxpbnV4L3N0YXQuaD4KICNpbmNsdWRlIDxsaW51 eC9jdHlwZS5oPgorI2luY2x1ZGUgPGxpbnV4L3BjaS5oPgorI2luY2x1ZGUgPGxpbnV4L3BjaS1w MnBkbWEuaD4KIAogI2luY2x1ZGUgIm52bWV0LmgiCiAKQEAgLTEwOTQsNiArMTA5NiwzNyBAQCBz dGF0aWMgdm9pZCBudm1ldF9wb3J0X3JlbGVhc2Uoc3RydWN0IGNvbmZpZ19pdGVtICppdGVtKQog CWtmcmVlKHBvcnQpOwogfQogCisjaWZkZWYgQ09ORklHX1BDSV9QMlBETUEKK3N0YXRpYyBzc2l6 ZV90IG52bWV0X3AycG1lbV9zaG93KHN0cnVjdCBjb25maWdfaXRlbSAqaXRlbSwgY2hhciAqcGFn ZSkKK3sKKwlzdHJ1Y3QgbnZtZXRfcG9ydCAqcG9ydCA9IHRvX252bWV0X3BvcnQoaXRlbSk7CisK KwlyZXR1cm4gcGNpX3AycGRtYV9lbmFibGVfc2hvdyhwYWdlLCBwb3J0LT5wMnBfZGV2LCBwb3J0 LT51c2VfcDJwbWVtKTsKK30KKworc3RhdGljIHNzaXplX3QgbnZtZXRfcDJwbWVtX3N0b3JlKHN0 cnVjdCBjb25maWdfaXRlbSAqaXRlbSwKKwkJY29uc3QgY2hhciAqcGFnZSwgc2l6ZV90IGNvdW50 KQoreworCXN0cnVjdCBudm1ldF9wb3J0ICpwb3J0ID0gdG9fbnZtZXRfcG9ydChpdGVtKTsKKwlz dHJ1Y3QgcGNpX2RldiAqcDJwX2RldiA9IE5VTEw7CisJYm9vbCB1c2VfcDJwbWVtOworCWludCBl cnJvcjsKKworCWVycm9yID0gcGNpX3AycGRtYV9lbmFibGVfc3RvcmUocGFnZSwgJnAycF9kZXYs ICZ1c2VfcDJwbWVtKTsKKwlpZiAoZXJyb3IpCisJCXJldHVybiBlcnJvcjsKKworCWRvd25fd3Jp dGUoJm52bWV0X2NvbmZpZ19zZW0pOworCXBvcnQtPnVzZV9wMnBtZW0gPSB1c2VfcDJwbWVtOwor CXBjaV9kZXZfcHV0KHBvcnQtPnAycF9kZXYpOworCXBvcnQtPnAycF9kZXYgPSBwMnBfZGV2Owor CXVwX3dyaXRlKCZudm1ldF9jb25maWdfc2VtKTsKKworCXJldHVybiBjb3VudDsKK30KK0NPTkZJ R0ZTX0FUVFIobnZtZXRfLCBwMnBtZW0pOworI2VuZGlmIC8qIENPTkZJR19QQ0lfUDJQRE1BICov CisKIHN0YXRpYyBzdHJ1Y3QgY29uZmlnZnNfYXR0cmlidXRlICpudm1ldF9wb3J0X2F0dHJzW10g PSB7CiAJJm52bWV0X2F0dHJfYWRkcl9hZHJmYW0sCiAJJm52bWV0X2F0dHJfYWRkcl90cmVxLApA QCAtMTEwMSw2ICsxMTM0LDkgQEAgc3RhdGljIHN0cnVjdCBjb25maWdmc19hdHRyaWJ1dGUgKm52 bWV0X3BvcnRfYXR0cnNbXSA9IHsKIAkmbnZtZXRfYXR0cl9hZGRyX3Ryc3ZjaWQsCiAJJm52bWV0 X2F0dHJfYWRkcl90cnR5cGUsCiAJJm52bWV0X2F0dHJfcGFyYW1faW5saW5lX2RhdGFfc2l6ZSwK KyNpZmRlZiBDT05GSUdfUENJX1AyUERNQQorCSZudm1ldF9hdHRyX3AycG1lbSwKKyNlbmRpZgog CU5VTEwsCiB9OwogCmRpZmYgLS1naXQgYS9kcml2ZXJzL252bWUvdGFyZ2V0L2NvcmUuYyBiL2Ry aXZlcnMvbnZtZS90YXJnZXQvY29yZS5jCmluZGV4IDZhMWM4ZDVmNTUyYi4uOGYyMGIxZTI2YzY5 IDEwMDY0NAotLS0gYS9kcml2ZXJzL252bWUvdGFyZ2V0L2NvcmUuYworKysgYi9kcml2ZXJzL252 bWUvdGFyZ2V0L2NvcmUuYwpAQCAtMTUsNiArMTUsNyBAQAogI2luY2x1ZGUgPGxpbnV4L21vZHVs ZS5oPgogI2luY2x1ZGUgPGxpbnV4L3JhbmRvbS5oPgogI2luY2x1ZGUgPGxpbnV4L3JjdWxpc3Qu aD4KKyNpbmNsdWRlIDxsaW51eC9wY2ktcDJwZG1hLmg+CiAKICNpbmNsdWRlICJudm1ldC5oIgog CkBAIC0zNjUsOSArMzY2LDI5IEBAIHN0YXRpYyB2b2lkIG52bWV0X25zX2Rldl9kaXNhYmxlKHN0 cnVjdCBudm1ldF9ucyAqbnMpCiAJbnZtZXRfZmlsZV9uc19kaXNhYmxlKG5zKTsKIH0KIAorc3Rh dGljIGludCBudm1ldF9wMnBkbWFfYWRkX2NsaWVudChzdHJ1Y3QgbnZtZXRfY3RybCAqY3RybCwK KwkJc3RydWN0IG52bWV0X25zICpucykKK3sKKwlpbnQgcmV0OworCisJaWYgKCFibGtfcXVldWVf cGNpX3AycGRtYShucy0+YmRldi0+YmRfcXVldWUpKSB7CisJCXByX2VycigicGVlci10by1wZWVy IERNQSBpcyBub3Qgc3VwcG9ydGVkIGJ5ICVzXG4iLAorCQkgICAgICAgbnMtPmRldmljZV9wYXRo KTsKKwkJcmV0dXJuIC1FSU5WQUw7CisJfQorCisJcmV0ID0gcGNpX3AycGRtYV9hZGRfY2xpZW50 KCZjdHJsLT5wMnBfY2xpZW50cywgbnZtZXRfbnNfZGV2KG5zKSk7CisJaWYgKHJldCkKKwkJcHJf ZXJyKCJmYWlsZWQgdG8gYWRkIHBlZXItdG8tcGVlciBETUEgY2xpZW50ICVzOiAlZFxuIiwKKwkJ ICAgICAgIG5zLT5kZXZpY2VfcGF0aCwgcmV0KTsKKworCXJldHVybiByZXQ7Cit9CisKIGludCBu dm1ldF9uc19lbmFibGUoc3RydWN0IG52bWV0X25zICpucykKIHsKIAlzdHJ1Y3QgbnZtZXRfc3Vi c3lzICpzdWJzeXMgPSBucy0+c3Vic3lzOworCXN0cnVjdCBudm1ldF9jdHJsICpjdHJsOwogCWlu dCByZXQ7CiAKIAltdXRleF9sb2NrKCZzdWJzeXMtPmxvY2spOwpAQCAtMzg5LDYgKzQxMCwxNCBA QCBpbnQgbnZtZXRfbnNfZW5hYmxlKHN0cnVjdCBudm1ldF9ucyAqbnMpCiAJaWYgKHJldCkKIAkJ Z290byBvdXRfZGV2X3B1dDsKIAorCWxpc3RfZm9yX2VhY2hfZW50cnkoY3RybCwgJnN1YnN5cy0+ Y3RybHMsIHN1YnN5c19lbnRyeSkgeworCQlpZiAoY3RybC0+cDJwX2RldikgeworCQkJcmV0ID0g bnZtZXRfcDJwZG1hX2FkZF9jbGllbnQoY3RybCwgbnMpOworCQkJaWYgKHJldCkKKwkJCQlnb3Rv IG91dF9yZW1vdmVfY2xpZW50czsKKwkJfQorCX0KKwogCWlmIChucy0+bnNpZCA+IHN1YnN5cy0+ bWF4X25zaWQpCiAJCXN1YnN5cy0+bWF4X25zaWQgPSBucy0+bnNpZDsKIApAQCAtNDE3LDYgKzQ0 Niw5IEBAIGludCBudm1ldF9uc19lbmFibGUoc3RydWN0IG52bWV0X25zICpucykKIG91dF91bmxv Y2s6CiAJbXV0ZXhfdW5sb2NrKCZzdWJzeXMtPmxvY2spOwogCXJldHVybiByZXQ7CitvdXRfcmVt b3ZlX2NsaWVudHM6CisJbGlzdF9mb3JfZWFjaF9lbnRyeShjdHJsLCAmc3Vic3lzLT5jdHJscywg c3Vic3lzX2VudHJ5KQorCQlwY2lfcDJwZG1hX3JlbW92ZV9jbGllbnQoJmN0cmwtPnAycF9jbGll bnRzLCBudm1ldF9uc19kZXYobnMpKTsKIG91dF9kZXZfcHV0OgogCW52bWV0X25zX2Rldl9kaXNh YmxlKG5zKTsKIAlnb3RvIG91dF91bmxvY2s7CkBAIC00MjUsNiArNDU3LDcgQEAgaW50IG52bWV0 X25zX2VuYWJsZShzdHJ1Y3QgbnZtZXRfbnMgKm5zKQogdm9pZCBudm1ldF9uc19kaXNhYmxlKHN0 cnVjdCBudm1ldF9ucyAqbnMpCiB7CiAJc3RydWN0IG52bWV0X3N1YnN5cyAqc3Vic3lzID0gbnMt PnN1YnN5czsKKwlzdHJ1Y3QgbnZtZXRfY3RybCAqY3RybDsKIAogCW11dGV4X2xvY2soJnN1YnN5 cy0+bG9jayk7CiAJaWYgKCFucy0+ZW5hYmxlZCkKQEAgLTQ1MCw2ICs0ODMsMTIgQEAgdm9pZCBu dm1ldF9uc19kaXNhYmxlKHN0cnVjdCBudm1ldF9ucyAqbnMpCiAJcGVyY3B1X3JlZl9leGl0KCZu cy0+cmVmKTsKIAogCW11dGV4X2xvY2soJnN1YnN5cy0+bG9jayk7CisKKwlsaXN0X2Zvcl9lYWNo X2VudHJ5KGN0cmwsICZzdWJzeXMtPmN0cmxzLCBzdWJzeXNfZW50cnkpIHsKKwkJcGNpX3AycGRt YV9yZW1vdmVfY2xpZW50KCZjdHJsLT5wMnBfY2xpZW50cywgbnZtZXRfbnNfZGV2KG5zKSk7CisJ CW52bWV0X2FkZF9hc3luY19ldmVudChjdHJsLCBOVk1FX0FFUl9UWVBFX05PVElDRSwgMCwgMCk7 CisJfQorCiAJc3Vic3lzLT5ucl9uYW1lc3BhY2VzLS07CiAJbnZtZXRfbnNfY2hhbmdlZChzdWJz eXMsIG5zLT5uc2lkKTsKIAludm1ldF9uc19kZXZfZGlzYWJsZShucyk7CkBAIC03MjcsNiArNzY2 LDIzIEBAIEVYUE9SVF9TWU1CT0xfR1BMKG52bWV0X3JlcV9leGVjdXRlKTsKIAogaW50IG52bWV0 X3JlcV9hbGxvY19zZ2woc3RydWN0IG52bWV0X3JlcSAqcmVxLCBzdHJ1Y3QgbnZtZXRfc3EgKnNx KQogeworCXN0cnVjdCBwY2lfZGV2ICpwMnBfZGV2ID0gTlVMTDsKKworCWlmIChJU19FTkFCTEVE KENPTkZJR19QQ0lfUDJQRE1BKSkgeworCQlpZiAoc3EtPmN0cmwpCisJCQlwMnBfZGV2ID0gc3Et PmN0cmwtPnAycF9kZXY7CisKKwkJcmVxLT5wMnBfZGV2ID0gTlVMTDsKKwkJaWYgKHNxLT5xaWQg JiYgcDJwX2RldikgeworCQkJcmVxLT5zZyA9IHBjaV9wMnBtZW1fYWxsb2Nfc2dsKHAycF9kZXYs ICZyZXEtPnNnX2NudCwKKwkJCQkJCSAgICAgICByZXEtPnRyYW5zZmVyX2xlbik7CisJCQlpZiAo cmVxLT5zZykgeworCQkJCXJlcS0+cDJwX2RldiA9IHAycF9kZXY7CisJCQkJcmV0dXJuIDA7CisJ CQl9CisJCX0KKwl9CisKIAlyZXEtPnNnID0gc2dsX2FsbG9jKHJlcS0+dHJhbnNmZXJfbGVuLCBH RlBfS0VSTkVMLCAmcmVxLT5zZ19jbnQpOwogCWlmICghcmVxLT5zZykKIAkJcmV0dXJuIC1FTk9N RU07CkBAIC03MzcsNyArNzkzLDExIEBAIEVYUE9SVF9TWU1CT0xfR1BMKG52bWV0X3JlcV9hbGxv Y19zZ2wpOwogCiB2b2lkIG52bWV0X3JlcV9mcmVlX3NnbChzdHJ1Y3QgbnZtZXRfcmVxICpyZXEp CiB7Ci0Jc2dsX2ZyZWUocmVxLT5zZyk7CisJaWYgKHJlcS0+cDJwX2RldikKKwkJcGNpX3AycG1l bV9mcmVlX3NnbChyZXEtPnAycF9kZXYsIHJlcS0+c2cpOworCWVsc2UKKwkJc2dsX2ZyZWUocmVx LT5zZyk7CisKIAlyZXEtPnNnID0gTlVMTDsKIAlyZXEtPnNnX2NudCA9IDA7CiB9CkBAIC05Mzks NiArOTk5LDc0IEBAIGJvb2wgbnZtZXRfaG9zdF9hbGxvd2VkKHN0cnVjdCBudm1ldF9yZXEgKnJl cSwgc3RydWN0IG52bWV0X3N1YnN5cyAqc3Vic3lzLAogCQlyZXR1cm4gX19udm1ldF9ob3N0X2Fs bG93ZWQoc3Vic3lzLCBob3N0bnFuKTsKIH0KIAorLyoKKyAqIElmIGFsbG93X3AycG1lbSBpcyBz ZXQsIHdlIHdpbGwgdHJ5IHRvIHVzZSBQMlAgbWVtb3J5IGZvciB0aGUgU0dMIGxpc3RzIGZvcgor ICogzpkvTyBjb21tYW5kcy4gVGhpcyByZXF1aXJlcyB0aGUgUENJIHAycCBkZXZpY2UgdG8gYmUg Y29tcGF0aWJsZSB3aXRoIHRoZQorICogYmFja2luZyBkZXZpY2UgZm9yIGV2ZXJ5IG5hbWVzcGFj ZSBvbiB0aGlzIGNvbnRyb2xsZXIuCisgKi8KK3N0YXRpYyB2b2lkIG52bWV0X3NldHVwX3AycG1l bShzdHJ1Y3QgbnZtZXRfY3RybCAqY3RybCwgc3RydWN0IG52bWV0X3JlcSAqcmVxKQoreworCXN0 cnVjdCBudm1ldF9ucyAqbnM7CisJaW50IHJldDsKKworCWlmICghcmVxLT5wb3J0LT51c2VfcDJw bWVtIHx8ICFyZXEtPnAycF9jbGllbnQpCisJCXJldHVybjsKKworCW11dGV4X2xvY2soJmN0cmwt PnN1YnN5cy0+bG9jayk7CisKKwlyZXQgPSBwY2lfcDJwZG1hX2FkZF9jbGllbnQoJmN0cmwtPnAy cF9jbGllbnRzLCByZXEtPnAycF9jbGllbnQpOworCWlmIChyZXQpIHsKKwkJcHJfZXJyKCJmYWls ZWQgYWRkaW5nIHBlZXItdG8tcGVlciBETUEgY2xpZW50ICVzOiAlZFxuIiwKKwkJICAgICAgIGRl dl9uYW1lKHJlcS0+cDJwX2NsaWVudCksIHJldCk7CisJCWdvdG8gZnJlZV9kZXZpY2VzOworCX0K KworCWxpc3RfZm9yX2VhY2hfZW50cnlfcmN1KG5zLCAmY3RybC0+c3Vic3lzLT5uYW1lc3BhY2Vz LCBkZXZfbGluaykgeworCQlyZXQgPSBudm1ldF9wMnBkbWFfYWRkX2NsaWVudChjdHJsLCBucyk7 CisJCWlmIChyZXQpCisJCQlnb3RvIGZyZWVfZGV2aWNlczsKKwl9CisKKwlpZiAocmVxLT5wb3J0 LT5wMnBfZGV2KSB7CisJCWlmICghcGNpX3AycGRtYV9hc3NpZ25fcHJvdmlkZXIocmVxLT5wb3J0 LT5wMnBfZGV2LAorCQkJCQkJJmN0cmwtPnAycF9jbGllbnRzKSkgeworCQkJcHJfaW5mbygicGVl ci10by1wZWVyIG1lbW9yeSBvbiAlcyBpcyBub3Qgc3VwcG9ydGVkXG4iLAorCQkJCXBjaV9uYW1l KHJlcS0+cG9ydC0+cDJwX2RldikpOworCQkJZ290byBmcmVlX2RldmljZXM7CisJCX0KKwkJY3Ry bC0+cDJwX2RldiA9IHBjaV9kZXZfZ2V0KHJlcS0+cG9ydC0+cDJwX2Rldik7CisJfSBlbHNlIHsK KwkJY3RybC0+cDJwX2RldiA9IHBjaV9wMnBtZW1fZmluZCgmY3RybC0+cDJwX2NsaWVudHMpOwor CQlpZiAoIWN0cmwtPnAycF9kZXYpIHsKKwkJCXByX2luZm8oIm5vIHN1cHBvcnRlZCBwZWVyLXRv LXBlZXIgbWVtb3J5IGRldmljZXMgZm91bmRcbiIpOworCQkJZ290byBmcmVlX2RldmljZXM7CisJ CX0KKwl9CisKKwltdXRleF91bmxvY2soJmN0cmwtPnN1YnN5cy0+bG9jayk7CisKKwlwcl9pbmZv KCJ1c2luZyBwZWVyLXRvLXBlZXIgbWVtb3J5IG9uICVzXG4iLCBwY2lfbmFtZShjdHJsLT5wMnBf ZGV2KSk7CisJcmV0dXJuOworCitmcmVlX2RldmljZXM6CisJcGNpX3AycGRtYV9jbGllbnRfbGlz dF9mcmVlKCZjdHJsLT5wMnBfY2xpZW50cyk7CisJbXV0ZXhfdW5sb2NrKCZjdHJsLT5zdWJzeXMt PmxvY2spOworfQorCitzdGF0aWMgdm9pZCBudm1ldF9yZWxlYXNlX3AycG1lbShzdHJ1Y3QgbnZt ZXRfY3RybCAqY3RybCkKK3sKKwlpZiAoIWN0cmwtPnAycF9kZXYpCisJCXJldHVybjsKKworCW11 dGV4X2xvY2soJmN0cmwtPnN1YnN5cy0+bG9jayk7CisKKwlwY2lfcDJwZG1hX2NsaWVudF9saXN0 X2ZyZWUoJmN0cmwtPnAycF9jbGllbnRzKTsKKwlwY2lfZGV2X3B1dChjdHJsLT5wMnBfZGV2KTsK KwljdHJsLT5wMnBfZGV2ID0gTlVMTDsKKworCW11dGV4X3VubG9jaygmY3RybC0+c3Vic3lzLT5s b2NrKTsKK30KKwogdTE2IG52bWV0X2FsbG9jX2N0cmwoY29uc3QgY2hhciAqc3Vic3lzbnFuLCBj b25zdCBjaGFyICpob3N0bnFuLAogCQlzdHJ1Y3QgbnZtZXRfcmVxICpyZXEsIHUzMiBrYXRvLCBz dHJ1Y3QgbnZtZXRfY3RybCAqKmN0cmxwKQogewpAQCAtOTgwLDYgKzExMDgsNyBAQCB1MTYgbnZt ZXRfYWxsb2NfY3RybChjb25zdCBjaGFyICpzdWJzeXNucW4sIGNvbnN0IGNoYXIgKmhvc3RucW4s CiAKIAlJTklUX1dPUksoJmN0cmwtPmFzeW5jX2V2ZW50X3dvcmssIG52bWV0X2FzeW5jX2V2ZW50 X3dvcmspOwogCUlOSVRfTElTVF9IRUFEKCZjdHJsLT5hc3luY19ldmVudHMpOworCUlOSVRfTElT VF9IRUFEKCZjdHJsLT5wMnBfY2xpZW50cyk7CiAKIAltZW1jcHkoY3RybC0+c3Vic3lzbnFuLCBz dWJzeXNucW4sIE5WTUZfTlFOX1NJWkUpOwogCW1lbWNweShjdHJsLT5ob3N0bnFuLCBob3N0bnFu LCBOVk1GX05RTl9TSVpFKTsKQEAgLTEwNDEsNiArMTE3MCw3IEBAIHUxNiBudm1ldF9hbGxvY19j dHJsKGNvbnN0IGNoYXIgKnN1YnN5c25xbiwgY29uc3QgY2hhciAqaG9zdG5xbiwKIAkJY3RybC0+ a2F0byA9IERJVl9ST1VORF9VUChrYXRvLCAxMDAwKTsKIAl9CiAJbnZtZXRfc3RhcnRfa2VlcF9h bGl2ZV90aW1lcihjdHJsKTsKKwludm1ldF9zZXR1cF9wMnBtZW0oY3RybCwgcmVxKTsKIAogCW11 dGV4X2xvY2soJnN1YnN5cy0+bG9jayk7CiAJbGlzdF9hZGRfdGFpbCgmY3RybC0+c3Vic3lzX2Vu dHJ5LCAmc3Vic3lzLT5jdHJscyk7CkBAIC0xMDc5LDYgKzEyMDksNyBAQCBzdGF0aWMgdm9pZCBu dm1ldF9jdHJsX2ZyZWUoc3RydWN0IGtyZWYgKnJlZikKIAlmbHVzaF93b3JrKCZjdHJsLT5hc3lu Y19ldmVudF93b3JrKTsKIAljYW5jZWxfd29ya19zeW5jKCZjdHJsLT5mYXRhbF9lcnJfd29yayk7 CiAKKwludm1ldF9yZWxlYXNlX3AycG1lbShjdHJsKTsKIAlpZGFfc2ltcGxlX3JlbW92ZSgmY250 bGlkX2lkYSwgY3RybC0+Y250bGlkKTsKIAogCWtmcmVlKGN0cmwtPnNxcyk7CmRpZmYgLS1naXQg YS9kcml2ZXJzL252bWUvdGFyZ2V0L252bWV0LmggYi9kcml2ZXJzL252bWUvdGFyZ2V0L252bWV0 LmgKaW5kZXggN2Q2Y2I2MTAyMWU0Li4yOTc4NjEwNjRkZDggMTAwNjQ0Ci0tLSBhL2RyaXZlcnMv bnZtZS90YXJnZXQvbnZtZXQuaAorKysgYi9kcml2ZXJzL252bWUvdGFyZ2V0L252bWV0LmgKQEAg LTg0LDYgKzg0LDExIEBAIHN0YXRpYyBpbmxpbmUgc3RydWN0IG52bWV0X25zICp0b19udm1ldF9u cyhzdHJ1Y3QgY29uZmlnX2l0ZW0gKml0ZW0pCiAJcmV0dXJuIGNvbnRhaW5lcl9vZih0b19jb25m aWdfZ3JvdXAoaXRlbSksIHN0cnVjdCBudm1ldF9ucywgZ3JvdXApOwogfQogCitzdGF0aWMgaW5s aW5lIHN0cnVjdCBkZXZpY2UgKm52bWV0X25zX2RldihzdHJ1Y3QgbnZtZXRfbnMgKm5zKQorewor CXJldHVybiBkaXNrX3RvX2Rldihucy0+YmRldi0+YmRfZGlzayk7Cit9CisKIHN0cnVjdCBudm1l dF9jcSB7CiAJdTE2CQkJcWlkOwogCXUxNgkJCXNpemU7CkBAIC0xMzQsNiArMTM5LDggQEAgc3Ry dWN0IG52bWV0X3BvcnQgewogCXZvaWQJCQkJKnByaXY7CiAJYm9vbAkJCQllbmFibGVkOwogCWlu dAkJCQlpbmxpbmVfZGF0YV9zaXplOworCWJvb2wJCQkJdXNlX3AycG1lbTsKKwlzdHJ1Y3QgcGNp X2RldgkJCSpwMnBfZGV2OwogfTsKIAogc3RhdGljIGlubGluZSBzdHJ1Y3QgbnZtZXRfcG9ydCAq dG9fbnZtZXRfcG9ydChzdHJ1Y3QgY29uZmlnX2l0ZW0gKml0ZW0pCkBAIC0xODIsNiArMTg5LDkg QEAgc3RydWN0IG52bWV0X2N0cmwgewogCV9fbGUzMgkJCSpjaGFuZ2VkX25zX2xpc3Q7CiAJdTMy CQkJbnJfY2hhbmdlZF9uczsKIAorCXN0cnVjdCBwY2lfZGV2CQkqcDJwX2RldjsKKwlzdHJ1Y3Qg bGlzdF9oZWFkCXAycF9jbGllbnRzOworCiAJY2hhcgkJCXN1YnN5c25xbltOVk1GX05RTl9GSUVM RF9MRU5dOwogCWNoYXIJCQlob3N0bnFuW05WTUZfTlFOX0ZJRUxEX0xFTl07CiB9OwpAQCAtMjk0 LDYgKzMwNCw5IEBAIHN0cnVjdCBudm1ldF9yZXEgewogCiAJdm9pZCAoKmV4ZWN1dGUpKHN0cnVj dCBudm1ldF9yZXEgKnJlcSk7CiAJY29uc3Qgc3RydWN0IG52bWV0X2ZhYnJpY3Nfb3BzICpvcHM7 CisKKwlzdHJ1Y3QgcGNpX2RldiAqcDJwX2RldjsKKwlzdHJ1Y3QgZGV2aWNlICpwMnBfY2xpZW50 OwogfTsKIAogZXh0ZXJuIHN0cnVjdCB3b3JrcXVldWVfc3RydWN0ICpidWZmZXJlZF9pb193cTsK ZGlmZiAtLWdpdCBhL2RyaXZlcnMvbnZtZS90YXJnZXQvcmRtYS5jIGIvZHJpdmVycy9udm1lL3Rh cmdldC9yZG1hLmMKaW5kZXggZTE0OGRlZTcyYmE1Li41YzljYjc1MmUyZWQgMTAwNjQ0Ci0tLSBh L2RyaXZlcnMvbnZtZS90YXJnZXQvcmRtYS5jCisrKyBiL2RyaXZlcnMvbnZtZS90YXJnZXQvcmRt YS5jCkBAIC03MzUsNiArNzM1LDggQEAgc3RhdGljIHZvaWQgbnZtZXRfcmRtYV9oYW5kbGVfY29t bWFuZChzdHJ1Y3QgbnZtZXRfcmRtYV9xdWV1ZSAqcXVldWUsCiAJCWNtZC0+c2VuZF9zZ2UuYWRk ciwgY21kLT5zZW5kX3NnZS5sZW5ndGgsCiAJCURNQV9UT19ERVZJQ0UpOwogCisJY21kLT5yZXEu cDJwX2NsaWVudCA9ICZxdWV1ZS0+ZGV2LT5kZXZpY2UtPmRldjsKKwogCWlmICghbnZtZXRfcmVx X2luaXQoJmNtZC0+cmVxLCAmcXVldWUtPm52bWVfY3EsCiAJCQkmcXVldWUtPm52bWVfc3EsICZu dm1ldF9yZG1hX29wcykpCiAJCXJldHVybjsKLS0gCjIuMTEuMAoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTGludXgtbnZkaW1tIG1haWxpbmcgbGlzdApM aW51eC1udmRpbW1AbGlzdHMuMDEub3JnCmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlz dGluZm8vbGludXgtbnZkaW1tCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com ([207.54.116.67]:40098 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727339AbeH3W5m (ORCPT ); Thu, 30 Aug 2018 18:57:42 -0400 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Stephen Bates , Christoph Hellwig , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?q?Christian=20K=C3=B6nig?= , Logan Gunthorpe , Steve Wise Date: Thu, 30 Aug 2018 12:53:52 -0600 Message-Id: <20180830185352.3369-14-logang@deltatee.com> In-Reply-To: <20180830185352.3369-1-logang@deltatee.com> References: <20180830185352.3369-1-logang@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Subject: [PATCH v5 13/13] nvmet: Optionally use PCI P2P memory Sender: linux-block-owner@vger.kernel.org List-Id: linux-block@vger.kernel.org We create a configfs attribute in each nvme-fabrics target port to enable p2p memory use. When enabled, the port will only then use the p2p memory if a p2p memory device can be found which is behind the same switch hierarchy as the RDMA port and all the block devices in use. If the user enabled it and no devices are found, then the system will silently fall back on using regular memory. If appropriate, that port will allocate memory for the RDMA buffers for queues from the p2pmem device falling back to system memory should anything fail. Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would save an extra PCI transfer as the NVME card could just take the data out of it's own memory. However, at this time, only a limited number of cards with CMB buffers seem to be available. Signed-off-by: Stephen Bates Signed-off-by: Steve Wise [hch: partial rewrite of the initial code] Signed-off-by: Christoph Hellwig Signed-off-by: Logan Gunthorpe --- drivers/nvme/target/configfs.c | 36 +++++++++++ drivers/nvme/target/core.c | 133 ++++++++++++++++++++++++++++++++++++++++- drivers/nvme/target/nvmet.h | 13 ++++ drivers/nvme/target/rdma.c | 2 + 4 files changed, 183 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c index b37a8e3e3f80..0dfb0e0c3d21 100644 --- a/drivers/nvme/target/configfs.c +++ b/drivers/nvme/target/configfs.c @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include "nvmet.h" @@ -1094,6 +1096,37 @@ static void nvmet_port_release(struct config_item *item) kfree(port); } +#ifdef CONFIG_PCI_P2PDMA +static ssize_t nvmet_p2pmem_show(struct config_item *item, char *page) +{ + struct nvmet_port *port = to_nvmet_port(item); + + return pci_p2pdma_enable_show(page, port->p2p_dev, port->use_p2pmem); +} + +static ssize_t nvmet_p2pmem_store(struct config_item *item, + const char *page, size_t count) +{ + struct nvmet_port *port = to_nvmet_port(item); + struct pci_dev *p2p_dev = NULL; + bool use_p2pmem; + int error; + + error = pci_p2pdma_enable_store(page, &p2p_dev, &use_p2pmem); + if (error) + return error; + + down_write(&nvmet_config_sem); + port->use_p2pmem = use_p2pmem; + pci_dev_put(port->p2p_dev); + port->p2p_dev = p2p_dev; + up_write(&nvmet_config_sem); + + return count; +} +CONFIGFS_ATTR(nvmet_, p2pmem); +#endif /* CONFIG_PCI_P2PDMA */ + static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_adrfam, &nvmet_attr_addr_treq, @@ -1101,6 +1134,9 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_trsvcid, &nvmet_attr_addr_trtype, &nvmet_attr_param_inline_data_size, +#ifdef CONFIG_PCI_P2PDMA + &nvmet_attr_p2pmem, +#endif NULL, }; diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c index 6a1c8d5f552b..8f20b1e26c69 100644 --- a/drivers/nvme/target/core.c +++ b/drivers/nvme/target/core.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "nvmet.h" @@ -365,9 +366,29 @@ static void nvmet_ns_dev_disable(struct nvmet_ns *ns) nvmet_file_ns_disable(ns); } +static int nvmet_p2pdma_add_client(struct nvmet_ctrl *ctrl, + struct nvmet_ns *ns) +{ + int ret; + + if (!blk_queue_pci_p2pdma(ns->bdev->bd_queue)) { + pr_err("peer-to-peer DMA is not supported by %s\n", + ns->device_path); + return -EINVAL; + } + + ret = pci_p2pdma_add_client(&ctrl->p2p_clients, nvmet_ns_dev(ns)); + if (ret) + pr_err("failed to add peer-to-peer DMA client %s: %d\n", + ns->device_path, ret); + + return ret; +} + int nvmet_ns_enable(struct nvmet_ns *ns) { struct nvmet_subsys *subsys = ns->subsys; + struct nvmet_ctrl *ctrl; int ret; mutex_lock(&subsys->lock); @@ -389,6 +410,14 @@ int nvmet_ns_enable(struct nvmet_ns *ns) if (ret) goto out_dev_put; + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) { + if (ctrl->p2p_dev) { + ret = nvmet_p2pdma_add_client(ctrl, ns); + if (ret) + goto out_remove_clients; + } + } + if (ns->nsid > subsys->max_nsid) subsys->max_nsid = ns->nsid; @@ -417,6 +446,9 @@ int nvmet_ns_enable(struct nvmet_ns *ns) out_unlock: mutex_unlock(&subsys->lock); return ret; +out_remove_clients: + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) + pci_p2pdma_remove_client(&ctrl->p2p_clients, nvmet_ns_dev(ns)); out_dev_put: nvmet_ns_dev_disable(ns); goto out_unlock; @@ -425,6 +457,7 @@ int nvmet_ns_enable(struct nvmet_ns *ns) void nvmet_ns_disable(struct nvmet_ns *ns) { struct nvmet_subsys *subsys = ns->subsys; + struct nvmet_ctrl *ctrl; mutex_lock(&subsys->lock); if (!ns->enabled) @@ -450,6 +483,12 @@ void nvmet_ns_disable(struct nvmet_ns *ns) percpu_ref_exit(&ns->ref); mutex_lock(&subsys->lock); + + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) { + pci_p2pdma_remove_client(&ctrl->p2p_clients, nvmet_ns_dev(ns)); + nvmet_add_async_event(ctrl, NVME_AER_TYPE_NOTICE, 0, 0); + } + subsys->nr_namespaces--; nvmet_ns_changed(subsys, ns->nsid); nvmet_ns_dev_disable(ns); @@ -727,6 +766,23 @@ EXPORT_SYMBOL_GPL(nvmet_req_execute); int nvmet_req_alloc_sgl(struct nvmet_req *req, struct nvmet_sq *sq) { + struct pci_dev *p2p_dev = NULL; + + if (IS_ENABLED(CONFIG_PCI_P2PDMA)) { + if (sq->ctrl) + p2p_dev = sq->ctrl->p2p_dev; + + req->p2p_dev = NULL; + if (sq->qid && p2p_dev) { + req->sg = pci_p2pmem_alloc_sgl(p2p_dev, &req->sg_cnt, + req->transfer_len); + if (req->sg) { + req->p2p_dev = p2p_dev; + return 0; + } + } + } + req->sg = sgl_alloc(req->transfer_len, GFP_KERNEL, &req->sg_cnt); if (!req->sg) return -ENOMEM; @@ -737,7 +793,11 @@ EXPORT_SYMBOL_GPL(nvmet_req_alloc_sgl); void nvmet_req_free_sgl(struct nvmet_req *req) { - sgl_free(req->sg); + if (req->p2p_dev) + pci_p2pmem_free_sgl(req->p2p_dev, req->sg); + else + sgl_free(req->sg); + req->sg = NULL; req->sg_cnt = 0; } @@ -939,6 +999,74 @@ bool nvmet_host_allowed(struct nvmet_req *req, struct nvmet_subsys *subsys, return __nvmet_host_allowed(subsys, hostnqn); } +/* + * If allow_p2pmem is set, we will try to use P2P memory for the SGL lists for + * Ι/O commands. This requires the PCI p2p device to be compatible with the + * backing device for every namespace on this controller. + */ +static void nvmet_setup_p2pmem(struct nvmet_ctrl *ctrl, struct nvmet_req *req) +{ + struct nvmet_ns *ns; + int ret; + + if (!req->port->use_p2pmem || !req->p2p_client) + return; + + mutex_lock(&ctrl->subsys->lock); + + ret = pci_p2pdma_add_client(&ctrl->p2p_clients, req->p2p_client); + if (ret) { + pr_err("failed adding peer-to-peer DMA client %s: %d\n", + dev_name(req->p2p_client), ret); + goto free_devices; + } + + list_for_each_entry_rcu(ns, &ctrl->subsys->namespaces, dev_link) { + ret = nvmet_p2pdma_add_client(ctrl, ns); + if (ret) + goto free_devices; + } + + if (req->port->p2p_dev) { + if (!pci_p2pdma_assign_provider(req->port->p2p_dev, + &ctrl->p2p_clients)) { + pr_info("peer-to-peer memory on %s is not supported\n", + pci_name(req->port->p2p_dev)); + goto free_devices; + } + ctrl->p2p_dev = pci_dev_get(req->port->p2p_dev); + } else { + ctrl->p2p_dev = pci_p2pmem_find(&ctrl->p2p_clients); + if (!ctrl->p2p_dev) { + pr_info("no supported peer-to-peer memory devices found\n"); + goto free_devices; + } + } + + mutex_unlock(&ctrl->subsys->lock); + + pr_info("using peer-to-peer memory on %s\n", pci_name(ctrl->p2p_dev)); + return; + +free_devices: + pci_p2pdma_client_list_free(&ctrl->p2p_clients); + mutex_unlock(&ctrl->subsys->lock); +} + +static void nvmet_release_p2pmem(struct nvmet_ctrl *ctrl) +{ + if (!ctrl->p2p_dev) + return; + + mutex_lock(&ctrl->subsys->lock); + + pci_p2pdma_client_list_free(&ctrl->p2p_clients); + pci_dev_put(ctrl->p2p_dev); + ctrl->p2p_dev = NULL; + + mutex_unlock(&ctrl->subsys->lock); +} + u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, struct nvmet_req *req, u32 kato, struct nvmet_ctrl **ctrlp) { @@ -980,6 +1108,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, INIT_WORK(&ctrl->async_event_work, nvmet_async_event_work); INIT_LIST_HEAD(&ctrl->async_events); + INIT_LIST_HEAD(&ctrl->p2p_clients); memcpy(ctrl->subsysnqn, subsysnqn, NVMF_NQN_SIZE); memcpy(ctrl->hostnqn, hostnqn, NVMF_NQN_SIZE); @@ -1041,6 +1170,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, ctrl->kato = DIV_ROUND_UP(kato, 1000); } nvmet_start_keep_alive_timer(ctrl); + nvmet_setup_p2pmem(ctrl, req); mutex_lock(&subsys->lock); list_add_tail(&ctrl->subsys_entry, &subsys->ctrls); @@ -1079,6 +1209,7 @@ static void nvmet_ctrl_free(struct kref *ref) flush_work(&ctrl->async_event_work); cancel_work_sync(&ctrl->fatal_err_work); + nvmet_release_p2pmem(ctrl); ida_simple_remove(&cntlid_ida, ctrl->cntlid); kfree(ctrl->sqs); diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 7d6cb61021e4..297861064dd8 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -84,6 +84,11 @@ static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item) return container_of(to_config_group(item), struct nvmet_ns, group); } +static inline struct device *nvmet_ns_dev(struct nvmet_ns *ns) +{ + return disk_to_dev(ns->bdev->bd_disk); +} + struct nvmet_cq { u16 qid; u16 size; @@ -134,6 +139,8 @@ struct nvmet_port { void *priv; bool enabled; int inline_data_size; + bool use_p2pmem; + struct pci_dev *p2p_dev; }; static inline struct nvmet_port *to_nvmet_port(struct config_item *item) @@ -182,6 +189,9 @@ struct nvmet_ctrl { __le32 *changed_ns_list; u32 nr_changed_ns; + struct pci_dev *p2p_dev; + struct list_head p2p_clients; + char subsysnqn[NVMF_NQN_FIELD_LEN]; char hostnqn[NVMF_NQN_FIELD_LEN]; }; @@ -294,6 +304,9 @@ struct nvmet_req { void (*execute)(struct nvmet_req *req); const struct nvmet_fabrics_ops *ops; + + struct pci_dev *p2p_dev; + struct device *p2p_client; }; extern struct workqueue_struct *buffered_io_wq; diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index e148dee72ba5..5c9cb752e2ed 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -735,6 +735,8 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue, cmd->send_sge.addr, cmd->send_sge.length, DMA_TO_DEVICE); + cmd->req.p2p_client = &queue->dev->device->dev; + if (!nvmet_req_init(&cmd->req, &queue->nvme_cq, &queue->nvme_sq, &nvmet_rdma_ops)) return; -- 2.11.0 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: [PATCH v5 13/13] nvmet: Optionally use PCI P2P memory Date: Thu, 30 Aug 2018 12:53:52 -0600 Message-ID: <20180830185352.3369-14-logang@deltatee.com> References: <20180830185352.3369-1-logang@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20180830185352.3369-1-logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Cc: =?UTF-8?q?Christian=20K=C3=B6nig?= , Benjamin Herrenschmidt , Steve Wise , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org V2UgY3JlYXRlIGEgY29uZmlnZnMgYXR0cmlidXRlIGluIGVhY2ggbnZtZS1mYWJyaWNzIHRhcmdl dCBwb3J0IHRvCmVuYWJsZSBwMnAgbWVtb3J5IHVzZS4gV2hlbiBlbmFibGVkLCB0aGUgcG9ydCB3 aWxsIG9ubHkgdGhlbiB1c2UgdGhlCnAycCBtZW1vcnkgaWYgYSBwMnAgbWVtb3J5IGRldmljZSBj YW4gYmUgZm91bmQgd2hpY2ggaXMgYmVoaW5kIHRoZQpzYW1lIHN3aXRjaCBoaWVyYXJjaHkgYXMg dGhlIFJETUEgcG9ydCBhbmQgYWxsIHRoZSBibG9jayBkZXZpY2VzIGluCnVzZS4gSWYgdGhlIHVz ZXIgZW5hYmxlZCBpdCBhbmQgbm8gZGV2aWNlcyBhcmUgZm91bmQsIHRoZW4gdGhlIHN5c3RlbQp3 aWxsIHNpbGVudGx5IGZhbGwgYmFjayBvbiB1c2luZyByZWd1bGFyIG1lbW9yeS4KCklmIGFwcHJv cHJpYXRlLCB0aGF0IHBvcnQgd2lsbCBhbGxvY2F0ZSBtZW1vcnkgZm9yIHRoZSBSRE1BIGJ1ZmZl cnMKZm9yIHF1ZXVlcyBmcm9tIHRoZSBwMnBtZW0gZGV2aWNlIGZhbGxpbmcgYmFjayB0byBzeXN0 ZW0gbWVtb3J5IHNob3VsZAphbnl0aGluZyBmYWlsLgoKSWRlYWxseSwgd2UnZCB3YW50IHRvIHVz ZSBhbiBOVk1FIENNQiBidWZmZXIgYXMgcDJwIG1lbW9yeS4gVGhpcyB3b3VsZApzYXZlIGFuIGV4 dHJhIFBDSSB0cmFuc2ZlciBhcyB0aGUgTlZNRSBjYXJkIGNvdWxkIGp1c3QgdGFrZSB0aGUgZGF0 YQpvdXQgb2YgaXQncyBvd24gbWVtb3J5LiBIb3dldmVyLCBhdCB0aGlzIHRpbWUsIG9ubHkgYSBs aW1pdGVkIG51bWJlcgpvZiBjYXJkcyB3aXRoIENNQiBidWZmZXJzIHNlZW0gdG8gYmUgYXZhaWxh YmxlLgoKU2lnbmVkLW9mZi1ieTogU3RlcGhlbiBCYXRlcyA8c2JhdGVzQHJhaXRobGluLmNvbT4K U2lnbmVkLW9mZi1ieTogU3RldmUgV2lzZSA8c3dpc2VAb3BlbmdyaWRjb21wdXRpbmcuY29tPgpb aGNoOiBwYXJ0aWFsIHJld3JpdGUgb2YgdGhlIGluaXRpYWwgY29kZV0KU2lnbmVkLW9mZi1ieTog Q2hyaXN0b3BoIEhlbGx3aWcgPGhjaEBsc3QuZGU+ClNpZ25lZC1vZmYtYnk6IExvZ2FuIEd1bnRo b3JwZSA8bG9nYW5nQGRlbHRhdGVlLmNvbT4KLS0tCiBkcml2ZXJzL252bWUvdGFyZ2V0L2NvbmZp Z2ZzLmMgfCAgMzYgKysrKysrKysrKysKIGRyaXZlcnMvbnZtZS90YXJnZXQvY29yZS5jICAgICB8 IDEzMyArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrLQogZHJpdmVycy9u dm1lL3RhcmdldC9udm1ldC5oICAgIHwgIDEzICsrKysKIGRyaXZlcnMvbnZtZS90YXJnZXQvcmRt YS5jICAgICB8ICAgMiArCiA0IGZpbGVzIGNoYW5nZWQsIDE4MyBpbnNlcnRpb25zKCspLCAxIGRl bGV0aW9uKC0pCgpkaWZmIC0tZ2l0IGEvZHJpdmVycy9udm1lL3RhcmdldC9jb25maWdmcy5jIGIv ZHJpdmVycy9udm1lL3RhcmdldC9jb25maWdmcy5jCmluZGV4IGIzN2E4ZTNlM2Y4MC4uMGRmYjBl MGMzZDIxIDEwMDY0NAotLS0gYS9kcml2ZXJzL252bWUvdGFyZ2V0L2NvbmZpZ2ZzLmMKKysrIGIv ZHJpdmVycy9udm1lL3RhcmdldC9jb25maWdmcy5jCkBAIC0xNyw2ICsxNyw4IEBACiAjaW5jbHVk ZSA8bGludXgvc2xhYi5oPgogI2luY2x1ZGUgPGxpbnV4L3N0YXQuaD4KICNpbmNsdWRlIDxsaW51 eC9jdHlwZS5oPgorI2luY2x1ZGUgPGxpbnV4L3BjaS5oPgorI2luY2x1ZGUgPGxpbnV4L3BjaS1w MnBkbWEuaD4KIAogI2luY2x1ZGUgIm52bWV0LmgiCiAKQEAgLTEwOTQsNiArMTA5NiwzNyBAQCBz dGF0aWMgdm9pZCBudm1ldF9wb3J0X3JlbGVhc2Uoc3RydWN0IGNvbmZpZ19pdGVtICppdGVtKQog CWtmcmVlKHBvcnQpOwogfQogCisjaWZkZWYgQ09ORklHX1BDSV9QMlBETUEKK3N0YXRpYyBzc2l6 ZV90IG52bWV0X3AycG1lbV9zaG93KHN0cnVjdCBjb25maWdfaXRlbSAqaXRlbSwgY2hhciAqcGFn ZSkKK3sKKwlzdHJ1Y3QgbnZtZXRfcG9ydCAqcG9ydCA9IHRvX252bWV0X3BvcnQoaXRlbSk7CisK KwlyZXR1cm4gcGNpX3AycGRtYV9lbmFibGVfc2hvdyhwYWdlLCBwb3J0LT5wMnBfZGV2LCBwb3J0 LT51c2VfcDJwbWVtKTsKK30KKworc3RhdGljIHNzaXplX3QgbnZtZXRfcDJwbWVtX3N0b3JlKHN0 cnVjdCBjb25maWdfaXRlbSAqaXRlbSwKKwkJY29uc3QgY2hhciAqcGFnZSwgc2l6ZV90IGNvdW50 KQoreworCXN0cnVjdCBudm1ldF9wb3J0ICpwb3J0ID0gdG9fbnZtZXRfcG9ydChpdGVtKTsKKwlz dHJ1Y3QgcGNpX2RldiAqcDJwX2RldiA9IE5VTEw7CisJYm9vbCB1c2VfcDJwbWVtOworCWludCBl cnJvcjsKKworCWVycm9yID0gcGNpX3AycGRtYV9lbmFibGVfc3RvcmUocGFnZSwgJnAycF9kZXYs ICZ1c2VfcDJwbWVtKTsKKwlpZiAoZXJyb3IpCisJCXJldHVybiBlcnJvcjsKKworCWRvd25fd3Jp dGUoJm52bWV0X2NvbmZpZ19zZW0pOworCXBvcnQtPnVzZV9wMnBtZW0gPSB1c2VfcDJwbWVtOwor CXBjaV9kZXZfcHV0KHBvcnQtPnAycF9kZXYpOworCXBvcnQtPnAycF9kZXYgPSBwMnBfZGV2Owor CXVwX3dyaXRlKCZudm1ldF9jb25maWdfc2VtKTsKKworCXJldHVybiBjb3VudDsKK30KK0NPTkZJ R0ZTX0FUVFIobnZtZXRfLCBwMnBtZW0pOworI2VuZGlmIC8qIENPTkZJR19QQ0lfUDJQRE1BICov CisKIHN0YXRpYyBzdHJ1Y3QgY29uZmlnZnNfYXR0cmlidXRlICpudm1ldF9wb3J0X2F0dHJzW10g PSB7CiAJJm52bWV0X2F0dHJfYWRkcl9hZHJmYW0sCiAJJm52bWV0X2F0dHJfYWRkcl90cmVxLApA QCAtMTEwMSw2ICsxMTM0LDkgQEAgc3RhdGljIHN0cnVjdCBjb25maWdmc19hdHRyaWJ1dGUgKm52 bWV0X3BvcnRfYXR0cnNbXSA9IHsKIAkmbnZtZXRfYXR0cl9hZGRyX3Ryc3ZjaWQsCiAJJm52bWV0 X2F0dHJfYWRkcl90cnR5cGUsCiAJJm52bWV0X2F0dHJfcGFyYW1faW5saW5lX2RhdGFfc2l6ZSwK KyNpZmRlZiBDT05GSUdfUENJX1AyUERNQQorCSZudm1ldF9hdHRyX3AycG1lbSwKKyNlbmRpZgog CU5VTEwsCiB9OwogCmRpZmYgLS1naXQgYS9kcml2ZXJzL252bWUvdGFyZ2V0L2NvcmUuYyBiL2Ry aXZlcnMvbnZtZS90YXJnZXQvY29yZS5jCmluZGV4IDZhMWM4ZDVmNTUyYi4uOGYyMGIxZTI2YzY5 IDEwMDY0NAotLS0gYS9kcml2ZXJzL252bWUvdGFyZ2V0L2NvcmUuYworKysgYi9kcml2ZXJzL252 bWUvdGFyZ2V0L2NvcmUuYwpAQCAtMTUsNiArMTUsNyBAQAogI2luY2x1ZGUgPGxpbnV4L21vZHVs ZS5oPgogI2luY2x1ZGUgPGxpbnV4L3JhbmRvbS5oPgogI2luY2x1ZGUgPGxpbnV4L3JjdWxpc3Qu aD4KKyNpbmNsdWRlIDxsaW51eC9wY2ktcDJwZG1hLmg+CiAKICNpbmNsdWRlICJudm1ldC5oIgog CkBAIC0zNjUsOSArMzY2LDI5IEBAIHN0YXRpYyB2b2lkIG52bWV0X25zX2Rldl9kaXNhYmxlKHN0 cnVjdCBudm1ldF9ucyAqbnMpCiAJbnZtZXRfZmlsZV9uc19kaXNhYmxlKG5zKTsKIH0KIAorc3Rh dGljIGludCBudm1ldF9wMnBkbWFfYWRkX2NsaWVudChzdHJ1Y3QgbnZtZXRfY3RybCAqY3RybCwK KwkJc3RydWN0IG52bWV0X25zICpucykKK3sKKwlpbnQgcmV0OworCisJaWYgKCFibGtfcXVldWVf cGNpX3AycGRtYShucy0+YmRldi0+YmRfcXVldWUpKSB7CisJCXByX2VycigicGVlci10by1wZWVy IERNQSBpcyBub3Qgc3VwcG9ydGVkIGJ5ICVzXG4iLAorCQkgICAgICAgbnMtPmRldmljZV9wYXRo KTsKKwkJcmV0dXJuIC1FSU5WQUw7CisJfQorCisJcmV0ID0gcGNpX3AycGRtYV9hZGRfY2xpZW50 KCZjdHJsLT5wMnBfY2xpZW50cywgbnZtZXRfbnNfZGV2KG5zKSk7CisJaWYgKHJldCkKKwkJcHJf ZXJyKCJmYWlsZWQgdG8gYWRkIHBlZXItdG8tcGVlciBETUEgY2xpZW50ICVzOiAlZFxuIiwKKwkJ ICAgICAgIG5zLT5kZXZpY2VfcGF0aCwgcmV0KTsKKworCXJldHVybiByZXQ7Cit9CisKIGludCBu dm1ldF9uc19lbmFibGUoc3RydWN0IG52bWV0X25zICpucykKIHsKIAlzdHJ1Y3QgbnZtZXRfc3Vi c3lzICpzdWJzeXMgPSBucy0+c3Vic3lzOworCXN0cnVjdCBudm1ldF9jdHJsICpjdHJsOwogCWlu dCByZXQ7CiAKIAltdXRleF9sb2NrKCZzdWJzeXMtPmxvY2spOwpAQCAtMzg5LDYgKzQxMCwxNCBA QCBpbnQgbnZtZXRfbnNfZW5hYmxlKHN0cnVjdCBudm1ldF9ucyAqbnMpCiAJaWYgKHJldCkKIAkJ Z290byBvdXRfZGV2X3B1dDsKIAorCWxpc3RfZm9yX2VhY2hfZW50cnkoY3RybCwgJnN1YnN5cy0+ Y3RybHMsIHN1YnN5c19lbnRyeSkgeworCQlpZiAoY3RybC0+cDJwX2RldikgeworCQkJcmV0ID0g bnZtZXRfcDJwZG1hX2FkZF9jbGllbnQoY3RybCwgbnMpOworCQkJaWYgKHJldCkKKwkJCQlnb3Rv IG91dF9yZW1vdmVfY2xpZW50czsKKwkJfQorCX0KKwogCWlmIChucy0+bnNpZCA+IHN1YnN5cy0+ bWF4X25zaWQpCiAJCXN1YnN5cy0+bWF4X25zaWQgPSBucy0+bnNpZDsKIApAQCAtNDE3LDYgKzQ0 Niw5IEBAIGludCBudm1ldF9uc19lbmFibGUoc3RydWN0IG52bWV0X25zICpucykKIG91dF91bmxv Y2s6CiAJbXV0ZXhfdW5sb2NrKCZzdWJzeXMtPmxvY2spOwogCXJldHVybiByZXQ7CitvdXRfcmVt b3ZlX2NsaWVudHM6CisJbGlzdF9mb3JfZWFjaF9lbnRyeShjdHJsLCAmc3Vic3lzLT5jdHJscywg c3Vic3lzX2VudHJ5KQorCQlwY2lfcDJwZG1hX3JlbW92ZV9jbGllbnQoJmN0cmwtPnAycF9jbGll bnRzLCBudm1ldF9uc19kZXYobnMpKTsKIG91dF9kZXZfcHV0OgogCW52bWV0X25zX2Rldl9kaXNh YmxlKG5zKTsKIAlnb3RvIG91dF91bmxvY2s7CkBAIC00MjUsNiArNDU3LDcgQEAgaW50IG52bWV0 X25zX2VuYWJsZShzdHJ1Y3QgbnZtZXRfbnMgKm5zKQogdm9pZCBudm1ldF9uc19kaXNhYmxlKHN0 cnVjdCBudm1ldF9ucyAqbnMpCiB7CiAJc3RydWN0IG52bWV0X3N1YnN5cyAqc3Vic3lzID0gbnMt PnN1YnN5czsKKwlzdHJ1Y3QgbnZtZXRfY3RybCAqY3RybDsKIAogCW11dGV4X2xvY2soJnN1YnN5 cy0+bG9jayk7CiAJaWYgKCFucy0+ZW5hYmxlZCkKQEAgLTQ1MCw2ICs0ODMsMTIgQEAgdm9pZCBu dm1ldF9uc19kaXNhYmxlKHN0cnVjdCBudm1ldF9ucyAqbnMpCiAJcGVyY3B1X3JlZl9leGl0KCZu cy0+cmVmKTsKIAogCW11dGV4X2xvY2soJnN1YnN5cy0+bG9jayk7CisKKwlsaXN0X2Zvcl9lYWNo X2VudHJ5KGN0cmwsICZzdWJzeXMtPmN0cmxzLCBzdWJzeXNfZW50cnkpIHsKKwkJcGNpX3AycGRt YV9yZW1vdmVfY2xpZW50KCZjdHJsLT5wMnBfY2xpZW50cywgbnZtZXRfbnNfZGV2KG5zKSk7CisJ CW52bWV0X2FkZF9hc3luY19ldmVudChjdHJsLCBOVk1FX0FFUl9UWVBFX05PVElDRSwgMCwgMCk7 CisJfQorCiAJc3Vic3lzLT5ucl9uYW1lc3BhY2VzLS07CiAJbnZtZXRfbnNfY2hhbmdlZChzdWJz eXMsIG5zLT5uc2lkKTsKIAludm1ldF9uc19kZXZfZGlzYWJsZShucyk7CkBAIC03MjcsNiArNzY2 LDIzIEBAIEVYUE9SVF9TWU1CT0xfR1BMKG52bWV0X3JlcV9leGVjdXRlKTsKIAogaW50IG52bWV0 X3JlcV9hbGxvY19zZ2woc3RydWN0IG52bWV0X3JlcSAqcmVxLCBzdHJ1Y3QgbnZtZXRfc3EgKnNx KQogeworCXN0cnVjdCBwY2lfZGV2ICpwMnBfZGV2ID0gTlVMTDsKKworCWlmIChJU19FTkFCTEVE KENPTkZJR19QQ0lfUDJQRE1BKSkgeworCQlpZiAoc3EtPmN0cmwpCisJCQlwMnBfZGV2ID0gc3Et PmN0cmwtPnAycF9kZXY7CisKKwkJcmVxLT5wMnBfZGV2ID0gTlVMTDsKKwkJaWYgKHNxLT5xaWQg JiYgcDJwX2RldikgeworCQkJcmVxLT5zZyA9IHBjaV9wMnBtZW1fYWxsb2Nfc2dsKHAycF9kZXYs ICZyZXEtPnNnX2NudCwKKwkJCQkJCSAgICAgICByZXEtPnRyYW5zZmVyX2xlbik7CisJCQlpZiAo cmVxLT5zZykgeworCQkJCXJlcS0+cDJwX2RldiA9IHAycF9kZXY7CisJCQkJcmV0dXJuIDA7CisJ CQl9CisJCX0KKwl9CisKIAlyZXEtPnNnID0gc2dsX2FsbG9jKHJlcS0+dHJhbnNmZXJfbGVuLCBH RlBfS0VSTkVMLCAmcmVxLT5zZ19jbnQpOwogCWlmICghcmVxLT5zZykKIAkJcmV0dXJuIC1FTk9N RU07CkBAIC03MzcsNyArNzkzLDExIEBAIEVYUE9SVF9TWU1CT0xfR1BMKG52bWV0X3JlcV9hbGxv Y19zZ2wpOwogCiB2b2lkIG52bWV0X3JlcV9mcmVlX3NnbChzdHJ1Y3QgbnZtZXRfcmVxICpyZXEp CiB7Ci0Jc2dsX2ZyZWUocmVxLT5zZyk7CisJaWYgKHJlcS0+cDJwX2RldikKKwkJcGNpX3AycG1l bV9mcmVlX3NnbChyZXEtPnAycF9kZXYsIHJlcS0+c2cpOworCWVsc2UKKwkJc2dsX2ZyZWUocmVx LT5zZyk7CisKIAlyZXEtPnNnID0gTlVMTDsKIAlyZXEtPnNnX2NudCA9IDA7CiB9CkBAIC05Mzks NiArOTk5LDc0IEBAIGJvb2wgbnZtZXRfaG9zdF9hbGxvd2VkKHN0cnVjdCBudm1ldF9yZXEgKnJl cSwgc3RydWN0IG52bWV0X3N1YnN5cyAqc3Vic3lzLAogCQlyZXR1cm4gX19udm1ldF9ob3N0X2Fs bG93ZWQoc3Vic3lzLCBob3N0bnFuKTsKIH0KIAorLyoKKyAqIElmIGFsbG93X3AycG1lbSBpcyBz ZXQsIHdlIHdpbGwgdHJ5IHRvIHVzZSBQMlAgbWVtb3J5IGZvciB0aGUgU0dMIGxpc3RzIGZvcgor ICogzpkvTyBjb21tYW5kcy4gVGhpcyByZXF1aXJlcyB0aGUgUENJIHAycCBkZXZpY2UgdG8gYmUg Y29tcGF0aWJsZSB3aXRoIHRoZQorICogYmFja2luZyBkZXZpY2UgZm9yIGV2ZXJ5IG5hbWVzcGFj ZSBvbiB0aGlzIGNvbnRyb2xsZXIuCisgKi8KK3N0YXRpYyB2b2lkIG52bWV0X3NldHVwX3AycG1l bShzdHJ1Y3QgbnZtZXRfY3RybCAqY3RybCwgc3RydWN0IG52bWV0X3JlcSAqcmVxKQoreworCXN0 cnVjdCBudm1ldF9ucyAqbnM7CisJaW50IHJldDsKKworCWlmICghcmVxLT5wb3J0LT51c2VfcDJw bWVtIHx8ICFyZXEtPnAycF9jbGllbnQpCisJCXJldHVybjsKKworCW11dGV4X2xvY2soJmN0cmwt PnN1YnN5cy0+bG9jayk7CisKKwlyZXQgPSBwY2lfcDJwZG1hX2FkZF9jbGllbnQoJmN0cmwtPnAy cF9jbGllbnRzLCByZXEtPnAycF9jbGllbnQpOworCWlmIChyZXQpIHsKKwkJcHJfZXJyKCJmYWls ZWQgYWRkaW5nIHBlZXItdG8tcGVlciBETUEgY2xpZW50ICVzOiAlZFxuIiwKKwkJICAgICAgIGRl dl9uYW1lKHJlcS0+cDJwX2NsaWVudCksIHJldCk7CisJCWdvdG8gZnJlZV9kZXZpY2VzOworCX0K KworCWxpc3RfZm9yX2VhY2hfZW50cnlfcmN1KG5zLCAmY3RybC0+c3Vic3lzLT5uYW1lc3BhY2Vz LCBkZXZfbGluaykgeworCQlyZXQgPSBudm1ldF9wMnBkbWFfYWRkX2NsaWVudChjdHJsLCBucyk7 CisJCWlmIChyZXQpCisJCQlnb3RvIGZyZWVfZGV2aWNlczsKKwl9CisKKwlpZiAocmVxLT5wb3J0 LT5wMnBfZGV2KSB7CisJCWlmICghcGNpX3AycGRtYV9hc3NpZ25fcHJvdmlkZXIocmVxLT5wb3J0 LT5wMnBfZGV2LAorCQkJCQkJJmN0cmwtPnAycF9jbGllbnRzKSkgeworCQkJcHJfaW5mbygicGVl ci10by1wZWVyIG1lbW9yeSBvbiAlcyBpcyBub3Qgc3VwcG9ydGVkXG4iLAorCQkJCXBjaV9uYW1l KHJlcS0+cG9ydC0+cDJwX2RldikpOworCQkJZ290byBmcmVlX2RldmljZXM7CisJCX0KKwkJY3Ry bC0+cDJwX2RldiA9IHBjaV9kZXZfZ2V0KHJlcS0+cG9ydC0+cDJwX2Rldik7CisJfSBlbHNlIHsK KwkJY3RybC0+cDJwX2RldiA9IHBjaV9wMnBtZW1fZmluZCgmY3RybC0+cDJwX2NsaWVudHMpOwor CQlpZiAoIWN0cmwtPnAycF9kZXYpIHsKKwkJCXByX2luZm8oIm5vIHN1cHBvcnRlZCBwZWVyLXRv LXBlZXIgbWVtb3J5IGRldmljZXMgZm91bmRcbiIpOworCQkJZ290byBmcmVlX2RldmljZXM7CisJ CX0KKwl9CisKKwltdXRleF91bmxvY2soJmN0cmwtPnN1YnN5cy0+bG9jayk7CisKKwlwcl9pbmZv KCJ1c2luZyBwZWVyLXRvLXBlZXIgbWVtb3J5IG9uICVzXG4iLCBwY2lfbmFtZShjdHJsLT5wMnBf ZGV2KSk7CisJcmV0dXJuOworCitmcmVlX2RldmljZXM6CisJcGNpX3AycGRtYV9jbGllbnRfbGlz dF9mcmVlKCZjdHJsLT5wMnBfY2xpZW50cyk7CisJbXV0ZXhfdW5sb2NrKCZjdHJsLT5zdWJzeXMt PmxvY2spOworfQorCitzdGF0aWMgdm9pZCBudm1ldF9yZWxlYXNlX3AycG1lbShzdHJ1Y3QgbnZt ZXRfY3RybCAqY3RybCkKK3sKKwlpZiAoIWN0cmwtPnAycF9kZXYpCisJCXJldHVybjsKKworCW11 dGV4X2xvY2soJmN0cmwtPnN1YnN5cy0+bG9jayk7CisKKwlwY2lfcDJwZG1hX2NsaWVudF9saXN0 X2ZyZWUoJmN0cmwtPnAycF9jbGllbnRzKTsKKwlwY2lfZGV2X3B1dChjdHJsLT5wMnBfZGV2KTsK KwljdHJsLT5wMnBfZGV2ID0gTlVMTDsKKworCW11dGV4X3VubG9jaygmY3RybC0+c3Vic3lzLT5s b2NrKTsKK30KKwogdTE2IG52bWV0X2FsbG9jX2N0cmwoY29uc3QgY2hhciAqc3Vic3lzbnFuLCBj b25zdCBjaGFyICpob3N0bnFuLAogCQlzdHJ1Y3QgbnZtZXRfcmVxICpyZXEsIHUzMiBrYXRvLCBz dHJ1Y3QgbnZtZXRfY3RybCAqKmN0cmxwKQogewpAQCAtOTgwLDYgKzExMDgsNyBAQCB1MTYgbnZt ZXRfYWxsb2NfY3RybChjb25zdCBjaGFyICpzdWJzeXNucW4sIGNvbnN0IGNoYXIgKmhvc3RucW4s CiAKIAlJTklUX1dPUksoJmN0cmwtPmFzeW5jX2V2ZW50X3dvcmssIG52bWV0X2FzeW5jX2V2ZW50 X3dvcmspOwogCUlOSVRfTElTVF9IRUFEKCZjdHJsLT5hc3luY19ldmVudHMpOworCUlOSVRfTElT VF9IRUFEKCZjdHJsLT5wMnBfY2xpZW50cyk7CiAKIAltZW1jcHkoY3RybC0+c3Vic3lzbnFuLCBz dWJzeXNucW4sIE5WTUZfTlFOX1NJWkUpOwogCW1lbWNweShjdHJsLT5ob3N0bnFuLCBob3N0bnFu LCBOVk1GX05RTl9TSVpFKTsKQEAgLTEwNDEsNiArMTE3MCw3IEBAIHUxNiBudm1ldF9hbGxvY19j dHJsKGNvbnN0IGNoYXIgKnN1YnN5c25xbiwgY29uc3QgY2hhciAqaG9zdG5xbiwKIAkJY3RybC0+ a2F0byA9IERJVl9ST1VORF9VUChrYXRvLCAxMDAwKTsKIAl9CiAJbnZtZXRfc3RhcnRfa2VlcF9h bGl2ZV90aW1lcihjdHJsKTsKKwludm1ldF9zZXR1cF9wMnBtZW0oY3RybCwgcmVxKTsKIAogCW11 dGV4X2xvY2soJnN1YnN5cy0+bG9jayk7CiAJbGlzdF9hZGRfdGFpbCgmY3RybC0+c3Vic3lzX2Vu dHJ5LCAmc3Vic3lzLT5jdHJscyk7CkBAIC0xMDc5LDYgKzEyMDksNyBAQCBzdGF0aWMgdm9pZCBu dm1ldF9jdHJsX2ZyZWUoc3RydWN0IGtyZWYgKnJlZikKIAlmbHVzaF93b3JrKCZjdHJsLT5hc3lu Y19ldmVudF93b3JrKTsKIAljYW5jZWxfd29ya19zeW5jKCZjdHJsLT5mYXRhbF9lcnJfd29yayk7 CiAKKwludm1ldF9yZWxlYXNlX3AycG1lbShjdHJsKTsKIAlpZGFfc2ltcGxlX3JlbW92ZSgmY250 bGlkX2lkYSwgY3RybC0+Y250bGlkKTsKIAogCWtmcmVlKGN0cmwtPnNxcyk7CmRpZmYgLS1naXQg YS9kcml2ZXJzL252bWUvdGFyZ2V0L252bWV0LmggYi9kcml2ZXJzL252bWUvdGFyZ2V0L252bWV0 LmgKaW5kZXggN2Q2Y2I2MTAyMWU0Li4yOTc4NjEwNjRkZDggMTAwNjQ0Ci0tLSBhL2RyaXZlcnMv bnZtZS90YXJnZXQvbnZtZXQuaAorKysgYi9kcml2ZXJzL252bWUvdGFyZ2V0L252bWV0LmgKQEAg LTg0LDYgKzg0LDExIEBAIHN0YXRpYyBpbmxpbmUgc3RydWN0IG52bWV0X25zICp0b19udm1ldF9u cyhzdHJ1Y3QgY29uZmlnX2l0ZW0gKml0ZW0pCiAJcmV0dXJuIGNvbnRhaW5lcl9vZih0b19jb25m aWdfZ3JvdXAoaXRlbSksIHN0cnVjdCBudm1ldF9ucywgZ3JvdXApOwogfQogCitzdGF0aWMgaW5s aW5lIHN0cnVjdCBkZXZpY2UgKm52bWV0X25zX2RldihzdHJ1Y3QgbnZtZXRfbnMgKm5zKQorewor CXJldHVybiBkaXNrX3RvX2Rldihucy0+YmRldi0+YmRfZGlzayk7Cit9CisKIHN0cnVjdCBudm1l dF9jcSB7CiAJdTE2CQkJcWlkOwogCXUxNgkJCXNpemU7CkBAIC0xMzQsNiArMTM5LDggQEAgc3Ry dWN0IG52bWV0X3BvcnQgewogCXZvaWQJCQkJKnByaXY7CiAJYm9vbAkJCQllbmFibGVkOwogCWlu dAkJCQlpbmxpbmVfZGF0YV9zaXplOworCWJvb2wJCQkJdXNlX3AycG1lbTsKKwlzdHJ1Y3QgcGNp X2RldgkJCSpwMnBfZGV2OwogfTsKIAogc3RhdGljIGlubGluZSBzdHJ1Y3QgbnZtZXRfcG9ydCAq dG9fbnZtZXRfcG9ydChzdHJ1Y3QgY29uZmlnX2l0ZW0gKml0ZW0pCkBAIC0xODIsNiArMTg5LDkg QEAgc3RydWN0IG52bWV0X2N0cmwgewogCV9fbGUzMgkJCSpjaGFuZ2VkX25zX2xpc3Q7CiAJdTMy CQkJbnJfY2hhbmdlZF9uczsKIAorCXN0cnVjdCBwY2lfZGV2CQkqcDJwX2RldjsKKwlzdHJ1Y3Qg bGlzdF9oZWFkCXAycF9jbGllbnRzOworCiAJY2hhcgkJCXN1YnN5c25xbltOVk1GX05RTl9GSUVM RF9MRU5dOwogCWNoYXIJCQlob3N0bnFuW05WTUZfTlFOX0ZJRUxEX0xFTl07CiB9OwpAQCAtMjk0 LDYgKzMwNCw5IEBAIHN0cnVjdCBudm1ldF9yZXEgewogCiAJdm9pZCAoKmV4ZWN1dGUpKHN0cnVj dCBudm1ldF9yZXEgKnJlcSk7CiAJY29uc3Qgc3RydWN0IG52bWV0X2ZhYnJpY3Nfb3BzICpvcHM7 CisKKwlzdHJ1Y3QgcGNpX2RldiAqcDJwX2RldjsKKwlzdHJ1Y3QgZGV2aWNlICpwMnBfY2xpZW50 OwogfTsKIAogZXh0ZXJuIHN0cnVjdCB3b3JrcXVldWVfc3RydWN0ICpidWZmZXJlZF9pb193cTsK ZGlmZiAtLWdpdCBhL2RyaXZlcnMvbnZtZS90YXJnZXQvcmRtYS5jIGIvZHJpdmVycy9udm1lL3Rh cmdldC9yZG1hLmMKaW5kZXggZTE0OGRlZTcyYmE1Li41YzljYjc1MmUyZWQgMTAwNjQ0Ci0tLSBh L2RyaXZlcnMvbnZtZS90YXJnZXQvcmRtYS5jCisrKyBiL2RyaXZlcnMvbnZtZS90YXJnZXQvcmRt YS5jCkBAIC03MzUsNiArNzM1LDggQEAgc3RhdGljIHZvaWQgbnZtZXRfcmRtYV9oYW5kbGVfY29t bWFuZChzdHJ1Y3QgbnZtZXRfcmRtYV9xdWV1ZSAqcXVldWUsCiAJCWNtZC0+c2VuZF9zZ2UuYWRk ciwgY21kLT5zZW5kX3NnZS5sZW5ndGgsCiAJCURNQV9UT19ERVZJQ0UpOwogCisJY21kLT5yZXEu cDJwX2NsaWVudCA9ICZxdWV1ZS0+ZGV2LT5kZXZpY2UtPmRldjsKKwogCWlmICghbnZtZXRfcmVx X2luaXQoJmNtZC0+cmVxLCAmcXVldWUtPm52bWVfY3EsCiAJCQkmcXVldWUtPm52bWVfc3EsICZu dm1ldF9yZG1hX29wcykpCiAJCXJldHVybjsKLS0gCjIuMTEuMAoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTGludXgtbnZkaW1tIG1haWxpbmcgbGlzdApM aW51eC1udmRpbW1AbGlzdHMuMDEub3JnCmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlz dGluZm8vbGludXgtbnZkaW1tCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: logang@deltatee.com (Logan Gunthorpe) Date: Thu, 30 Aug 2018 12:53:52 -0600 Subject: [PATCH v5 13/13] nvmet: Optionally use PCI P2P memory In-Reply-To: <20180830185352.3369-1-logang@deltatee.com> References: <20180830185352.3369-1-logang@deltatee.com> Message-ID: <20180830185352.3369-14-logang@deltatee.com> We create a configfs attribute in each nvme-fabrics target port to enable p2p memory use. When enabled, the port will only then use the p2p memory if a p2p memory device can be found which is behind the same switch hierarchy as the RDMA port and all the block devices in use. If the user enabled it and no devices are found, then the system will silently fall back on using regular memory. If appropriate, that port will allocate memory for the RDMA buffers for queues from the p2pmem device falling back to system memory should anything fail. Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would save an extra PCI transfer as the NVME card could just take the data out of it's own memory. However, at this time, only a limited number of cards with CMB buffers seem to be available. Signed-off-by: Stephen Bates Signed-off-by: Steve Wise [hch: partial rewrite of the initial code] Signed-off-by: Christoph Hellwig Signed-off-by: Logan Gunthorpe --- drivers/nvme/target/configfs.c | 36 +++++++++++ drivers/nvme/target/core.c | 133 ++++++++++++++++++++++++++++++++++++++++- drivers/nvme/target/nvmet.h | 13 ++++ drivers/nvme/target/rdma.c | 2 + 4 files changed, 183 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c index b37a8e3e3f80..0dfb0e0c3d21 100644 --- a/drivers/nvme/target/configfs.c +++ b/drivers/nvme/target/configfs.c @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include "nvmet.h" @@ -1094,6 +1096,37 @@ static void nvmet_port_release(struct config_item *item) kfree(port); } +#ifdef CONFIG_PCI_P2PDMA +static ssize_t nvmet_p2pmem_show(struct config_item *item, char *page) +{ + struct nvmet_port *port = to_nvmet_port(item); + + return pci_p2pdma_enable_show(page, port->p2p_dev, port->use_p2pmem); +} + +static ssize_t nvmet_p2pmem_store(struct config_item *item, + const char *page, size_t count) +{ + struct nvmet_port *port = to_nvmet_port(item); + struct pci_dev *p2p_dev = NULL; + bool use_p2pmem; + int error; + + error = pci_p2pdma_enable_store(page, &p2p_dev, &use_p2pmem); + if (error) + return error; + + down_write(&nvmet_config_sem); + port->use_p2pmem = use_p2pmem; + pci_dev_put(port->p2p_dev); + port->p2p_dev = p2p_dev; + up_write(&nvmet_config_sem); + + return count; +} +CONFIGFS_ATTR(nvmet_, p2pmem); +#endif /* CONFIG_PCI_P2PDMA */ + static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_adrfam, &nvmet_attr_addr_treq, @@ -1101,6 +1134,9 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_trsvcid, &nvmet_attr_addr_trtype, &nvmet_attr_param_inline_data_size, +#ifdef CONFIG_PCI_P2PDMA + &nvmet_attr_p2pmem, +#endif NULL, }; diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c index 6a1c8d5f552b..8f20b1e26c69 100644 --- a/drivers/nvme/target/core.c +++ b/drivers/nvme/target/core.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "nvmet.h" @@ -365,9 +366,29 @@ static void nvmet_ns_dev_disable(struct nvmet_ns *ns) nvmet_file_ns_disable(ns); } +static int nvmet_p2pdma_add_client(struct nvmet_ctrl *ctrl, + struct nvmet_ns *ns) +{ + int ret; + + if (!blk_queue_pci_p2pdma(ns->bdev->bd_queue)) { + pr_err("peer-to-peer DMA is not supported by %s\n", + ns->device_path); + return -EINVAL; + } + + ret = pci_p2pdma_add_client(&ctrl->p2p_clients, nvmet_ns_dev(ns)); + if (ret) + pr_err("failed to add peer-to-peer DMA client %s: %d\n", + ns->device_path, ret); + + return ret; +} + int nvmet_ns_enable(struct nvmet_ns *ns) { struct nvmet_subsys *subsys = ns->subsys; + struct nvmet_ctrl *ctrl; int ret; mutex_lock(&subsys->lock); @@ -389,6 +410,14 @@ int nvmet_ns_enable(struct nvmet_ns *ns) if (ret) goto out_dev_put; + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) { + if (ctrl->p2p_dev) { + ret = nvmet_p2pdma_add_client(ctrl, ns); + if (ret) + goto out_remove_clients; + } + } + if (ns->nsid > subsys->max_nsid) subsys->max_nsid = ns->nsid; @@ -417,6 +446,9 @@ int nvmet_ns_enable(struct nvmet_ns *ns) out_unlock: mutex_unlock(&subsys->lock); return ret; +out_remove_clients: + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) + pci_p2pdma_remove_client(&ctrl->p2p_clients, nvmet_ns_dev(ns)); out_dev_put: nvmet_ns_dev_disable(ns); goto out_unlock; @@ -425,6 +457,7 @@ int nvmet_ns_enable(struct nvmet_ns *ns) void nvmet_ns_disable(struct nvmet_ns *ns) { struct nvmet_subsys *subsys = ns->subsys; + struct nvmet_ctrl *ctrl; mutex_lock(&subsys->lock); if (!ns->enabled) @@ -450,6 +483,12 @@ void nvmet_ns_disable(struct nvmet_ns *ns) percpu_ref_exit(&ns->ref); mutex_lock(&subsys->lock); + + list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) { + pci_p2pdma_remove_client(&ctrl->p2p_clients, nvmet_ns_dev(ns)); + nvmet_add_async_event(ctrl, NVME_AER_TYPE_NOTICE, 0, 0); + } + subsys->nr_namespaces--; nvmet_ns_changed(subsys, ns->nsid); nvmet_ns_dev_disable(ns); @@ -727,6 +766,23 @@ EXPORT_SYMBOL_GPL(nvmet_req_execute); int nvmet_req_alloc_sgl(struct nvmet_req *req, struct nvmet_sq *sq) { + struct pci_dev *p2p_dev = NULL; + + if (IS_ENABLED(CONFIG_PCI_P2PDMA)) { + if (sq->ctrl) + p2p_dev = sq->ctrl->p2p_dev; + + req->p2p_dev = NULL; + if (sq->qid && p2p_dev) { + req->sg = pci_p2pmem_alloc_sgl(p2p_dev, &req->sg_cnt, + req->transfer_len); + if (req->sg) { + req->p2p_dev = p2p_dev; + return 0; + } + } + } + req->sg = sgl_alloc(req->transfer_len, GFP_KERNEL, &req->sg_cnt); if (!req->sg) return -ENOMEM; @@ -737,7 +793,11 @@ EXPORT_SYMBOL_GPL(nvmet_req_alloc_sgl); void nvmet_req_free_sgl(struct nvmet_req *req) { - sgl_free(req->sg); + if (req->p2p_dev) + pci_p2pmem_free_sgl(req->p2p_dev, req->sg); + else + sgl_free(req->sg); + req->sg = NULL; req->sg_cnt = 0; } @@ -939,6 +999,74 @@ bool nvmet_host_allowed(struct nvmet_req *req, struct nvmet_subsys *subsys, return __nvmet_host_allowed(subsys, hostnqn); } +/* + * If allow_p2pmem is set, we will try to use P2P memory for the SGL lists for + * ?/O commands. This requires the PCI p2p device to be compatible with the + * backing device for every namespace on this controller. + */ +static void nvmet_setup_p2pmem(struct nvmet_ctrl *ctrl, struct nvmet_req *req) +{ + struct nvmet_ns *ns; + int ret; + + if (!req->port->use_p2pmem || !req->p2p_client) + return; + + mutex_lock(&ctrl->subsys->lock); + + ret = pci_p2pdma_add_client(&ctrl->p2p_clients, req->p2p_client); + if (ret) { + pr_err("failed adding peer-to-peer DMA client %s: %d\n", + dev_name(req->p2p_client), ret); + goto free_devices; + } + + list_for_each_entry_rcu(ns, &ctrl->subsys->namespaces, dev_link) { + ret = nvmet_p2pdma_add_client(ctrl, ns); + if (ret) + goto free_devices; + } + + if (req->port->p2p_dev) { + if (!pci_p2pdma_assign_provider(req->port->p2p_dev, + &ctrl->p2p_clients)) { + pr_info("peer-to-peer memory on %s is not supported\n", + pci_name(req->port->p2p_dev)); + goto free_devices; + } + ctrl->p2p_dev = pci_dev_get(req->port->p2p_dev); + } else { + ctrl->p2p_dev = pci_p2pmem_find(&ctrl->p2p_clients); + if (!ctrl->p2p_dev) { + pr_info("no supported peer-to-peer memory devices found\n"); + goto free_devices; + } + } + + mutex_unlock(&ctrl->subsys->lock); + + pr_info("using peer-to-peer memory on %s\n", pci_name(ctrl->p2p_dev)); + return; + +free_devices: + pci_p2pdma_client_list_free(&ctrl->p2p_clients); + mutex_unlock(&ctrl->subsys->lock); +} + +static void nvmet_release_p2pmem(struct nvmet_ctrl *ctrl) +{ + if (!ctrl->p2p_dev) + return; + + mutex_lock(&ctrl->subsys->lock); + + pci_p2pdma_client_list_free(&ctrl->p2p_clients); + pci_dev_put(ctrl->p2p_dev); + ctrl->p2p_dev = NULL; + + mutex_unlock(&ctrl->subsys->lock); +} + u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, struct nvmet_req *req, u32 kato, struct nvmet_ctrl **ctrlp) { @@ -980,6 +1108,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, INIT_WORK(&ctrl->async_event_work, nvmet_async_event_work); INIT_LIST_HEAD(&ctrl->async_events); + INIT_LIST_HEAD(&ctrl->p2p_clients); memcpy(ctrl->subsysnqn, subsysnqn, NVMF_NQN_SIZE); memcpy(ctrl->hostnqn, hostnqn, NVMF_NQN_SIZE); @@ -1041,6 +1170,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, ctrl->kato = DIV_ROUND_UP(kato, 1000); } nvmet_start_keep_alive_timer(ctrl); + nvmet_setup_p2pmem(ctrl, req); mutex_lock(&subsys->lock); list_add_tail(&ctrl->subsys_entry, &subsys->ctrls); @@ -1079,6 +1209,7 @@ static void nvmet_ctrl_free(struct kref *ref) flush_work(&ctrl->async_event_work); cancel_work_sync(&ctrl->fatal_err_work); + nvmet_release_p2pmem(ctrl); ida_simple_remove(&cntlid_ida, ctrl->cntlid); kfree(ctrl->sqs); diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 7d6cb61021e4..297861064dd8 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -84,6 +84,11 @@ static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item) return container_of(to_config_group(item), struct nvmet_ns, group); } +static inline struct device *nvmet_ns_dev(struct nvmet_ns *ns) +{ + return disk_to_dev(ns->bdev->bd_disk); +} + struct nvmet_cq { u16 qid; u16 size; @@ -134,6 +139,8 @@ struct nvmet_port { void *priv; bool enabled; int inline_data_size; + bool use_p2pmem; + struct pci_dev *p2p_dev; }; static inline struct nvmet_port *to_nvmet_port(struct config_item *item) @@ -182,6 +189,9 @@ struct nvmet_ctrl { __le32 *changed_ns_list; u32 nr_changed_ns; + struct pci_dev *p2p_dev; + struct list_head p2p_clients; + char subsysnqn[NVMF_NQN_FIELD_LEN]; char hostnqn[NVMF_NQN_FIELD_LEN]; }; @@ -294,6 +304,9 @@ struct nvmet_req { void (*execute)(struct nvmet_req *req); const struct nvmet_fabrics_ops *ops; + + struct pci_dev *p2p_dev; + struct device *p2p_client; }; extern struct workqueue_struct *buffered_io_wq; diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index e148dee72ba5..5c9cb752e2ed 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -735,6 +735,8 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue, cmd->send_sge.addr, cmd->send_sge.length, DMA_TO_DEVICE); + cmd->req.p2p_client = &queue->dev->device->dev; + if (!nvmet_req_init(&cmd->req, &queue->nvme_cq, &queue->nvme_sq, &nvmet_rdma_ops)) return; -- 2.11.0