From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kenneth Lee Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive Date: Tue, 11 Sep 2018 14:40:43 +0800 Message-ID: <20180911064043.GA207969@Turing-Arch-b> References: <20180904150019.GA4024@redhat.com> <20180904101509.62314b67@t450s.home> <20180906094532.GG230707@Turing-Arch-b> <20180906133133.GA3830@redhat.com> <20180907040138.GI230707@Turing-Arch-b> <20180907165303.GA3519@redhat.com> <20180910032809.GJ230707@Turing-Arch-b> <20180910145423.GA3488@redhat.com> <20180911024209.GK230707@Turing-Arch-b> <20180911033358.GA4730@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: Kenneth Lee , Alex Williamson , Herbert Xu , kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Jonathan Corbet , Greg Kroah-Hartman , linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Sanjay Kumar , Hao Fang , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linuxarm-hv44wF8Li93QT0dZR+AlfA@public.gmane.org, iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org, "David S . Miller" , linux-crypto-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Philippe Ombredanne , Thomas Gleixner , linux-accelerators-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org To: Jerome Glisse Return-path: Content-Disposition: inline In-Reply-To: <20180911033358.GA4730-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org List-Id: linux-crypto.vger.kernel.org T24gTW9uLCBTZXAgMTAsIDIwMTggYXQgMTE6MzM6NTlQTSAtMDQwMCwgSmVyb21lIEdsaXNzZSB3 cm90ZToKPiBEYXRlOiBNb24sIDEwIFNlcCAyMDE4IDIzOjMzOjU5IC0wNDAwCj4gRnJvbTogSmVy b21lIEdsaXNzZSA8amdsaXNzZUByZWRoYXQuY29tPgo+IFRvOiBLZW5uZXRoIExlZSA8bGlndW96 aHVAaGlzaWxpY29uLmNvbT4KPiBDQzogS2VubmV0aCBMZWUgPG5lay5pbi5jbkBnbWFpbC5jb20+ LCBaYWlibyBYdSA8eHV6YWlib0BodWF3ZWkuY29tPiwKPiAgSGVyYmVydCBYdSA8aGVyYmVydEBn b25kb3IuYXBhbmEub3JnLmF1Piwga3ZtQHZnZXIua2VybmVsLm9yZywgSm9uYXRoYW4KPiAgQ29y YmV0IDxjb3JiZXRAbHduLm5ldD4sIEdyZWcgS3JvYWgtSGFydG1hbiA8Z3JlZ2toQGxpbnV4Zm91 bmRhdGlvbi5vcmc+LAo+ICBKb2VyZyBSb2VkZWwgPGpvcm9AOGJ5dGVzLm9yZz4sIGxpbnV4LWRv Y0B2Z2VyLmtlcm5lbC5vcmcsIFNhbmpheSBLdW1hcgo+ICA8c2FuamF5Lmsua3VtYXJAaW50ZWwu Y29tPiwgSGFvIEZhbmcgPGZhbmdoYW8xMUBodWF3ZWkuY29tPiwKPiAgaW9tbXVAbGlzdHMubGlu dXgtZm91bmRhdGlvbi5vcmcsIGxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmcsCj4gIGxpbnV4 YXJtQGh1YXdlaS5jb20sIEFsZXggV2lsbGlhbXNvbiA8YWxleC53aWxsaWFtc29uQHJlZGhhdC5j b20+LAo+ICBsaW51eC1jcnlwdG9Admdlci5rZXJuZWwub3JnLCBaaG91IFdhbmcgPHdhbmd6aG91 MUBoaXNpbGljb24uY29tPiwKPiAgUGhpbGlwcGUgT21icmVkYW5uZSA8cG9tYnJlZGFubmVAbmV4 Yi5jb20+LCBUaG9tYXMgR2xlaXhuZXIKPiAgPHRnbHhAbGludXRyb25peC5kZT4sICJEYXZpZCBT IC4gTWlsbGVyIiA8ZGF2ZW1AZGF2ZW1sb2Z0Lm5ldD4sCj4gIGxpbnV4LWFjY2VsZXJhdG9yc0Bs aXN0cy5vemxhYnMub3JnLCBMdSBCYW9sdSA8YmFvbHUubHVAbGludXguaW50ZWwuY29tPgo+IFN1 YmplY3Q6IFJlOiBbUkZDdjIgUEFUQ0ggMC83XSBBIEdlbmVyYWwgQWNjZWxlcmF0b3IgRnJhbWV3 b3JrLCBXYXJwRHJpdmUKPiBVc2VyLUFnZW50OiBNdXR0LzEuMTAuMSAoMjAxOC0wNy0xMykKPiBN ZXNzYWdlLUlEOiA8MjAxODA5MTEwMzMzNTguR0E0NzMwQHJlZGhhdC5jb20+Cj4gCj4gT24gVHVl LCBTZXAgMTEsIDIwMTggYXQgMTA6NDI6MDlBTSArMDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4g PiBPbiBNb24sIFNlcCAxMCwgMjAxOCBhdCAxMDo1NDoyM0FNIC0wNDAwLCBKZXJvbWUgR2xpc3Nl IHdyb3RlOgo+ID4gPiBPbiBNb24sIFNlcCAxMCwgMjAxOCBhdCAxMToyODowOUFNICswODAwLCBL ZW5uZXRoIExlZSB3cm90ZToKPiA+ID4gPiBPbiBGcmksIFNlcCAwNywgMjAxOCBhdCAxMjo1Mzow NlBNIC0wNDAwLCBKZXJvbWUgR2xpc3NlIHdyb3RlOgo+ID4gPiA+ID4gT24gRnJpLCBTZXAgMDcs IDIwMTggYXQgMTI6MDE6MzhQTSArMDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4gPiA+ID4gPiA+ IE9uIFRodSwgU2VwIDA2LCAyMDE4IGF0IDA5OjMxOjMzQU0gLTA0MDAsIEplcm9tZSBHbGlzc2Ug d3JvdGU6Cj4gPiA+ID4gPiA+ID4gT24gVGh1LCBTZXAgMDYsIDIwMTggYXQgMDU6NDU6MzJQTSAr MDgwMCwgS2VubmV0aCBMZWUgd3JvdGU6Cj4gPiA+ID4gPiA+ID4gPiBPbiBUdWUsIFNlcCAwNCwg MjAxOCBhdCAxMDoxNTowOUFNIC0wNjAwLCBBbGV4IFdpbGxpYW1zb24gd3JvdGU6Cj4gPiA+ID4g PiA+ID4gPiA+IE9uIFR1ZSwgNCBTZXAgMjAxOCAxMTowMDoxOSAtMDQwMCBKZXJvbWUgR2xpc3Nl IDxqZ2xpc3NlQHJlZGhhdC5jb20+IHdyb3RlOgo+ID4gPiA+ID4gPiA+ID4gPiA+IE9uIE1vbiwg U2VwIDAzLCAyMDE4IGF0IDA4OjUxOjU3QU0gKzA4MDAsIEtlbm5ldGggTGVlIHdyb3RlOgo+ID4g PiAKPiA+ID4gWy4uLl0KPiA+ID4gCj4gPiA+ID4gPiA+IEkgdG9vayBhIGxvb2sgYXQgaTkxNV9n ZW1fZXhlY2J1ZmZlcl9pb2N0bCgpLiBJdCBzZWVtcyBpdCAiY29weV9mcm9tX3VzZXIiIHRoZQo+ ID4gPiA+ID4gPiB1c2VyIG1lbW9yeSB0byB0aGUga2VybmVsLiBUaGF0IGlzIG5vdCB3aGF0IHdl IG5lZWQuIFdoYXQgd2UgdHJ5IHRvIGdldCBpczogdGhlCj4gPiA+ID4gPiA+IHVzZXIgYXBwbGlj YXRpb24gZG8gc29tZXRoaW5nIG9uIGl0cyBkYXRhLCBhbmQgcHVzaCBpdCBhd2F5IHRvIHRoZSBh Y2NlbGVyYXRvciwKPiA+ID4gPiA+ID4gYW5kIHNheXM6ICJJJ20gdGllZCwgaXQgaXMgeW91ciB0 dXJuIHRvIGRvIHRoZSBqb2IuLi4iLiBUaGVuIHRoZSBhY2NlbGVyYXRvciBoYXMKPiA+ID4gPiA+ ID4gdGhlIG1lbW9yeSwgcmVmZXJyaW5nIGFueSBwb3J0aW9uIG9mIGl0IHdpdGggdGhlIHNhbWUg VkFzIG9mIHRoZSBhcHBsaWNhdGlvbiwKPiA+ID4gPiA+ID4gZXZlbiB0aGUgVkFzIGFyZSBzdG9y ZWQgaW5zaWRlIHRoZSBtZW1vcnkgaXRzZWxmLgo+ID4gPiA+ID4gCj4gPiA+ID4gPiBZb3Ugd2Vy ZSBub3QgbG9va2luZyBhdCByaWdodCBwbGFjZSBzZWUgZHJpdmVycy9ncHUvZHJtL2k5MTUvaTkx NV9nZW1fdXNlcnB0ci5jCj4gPiA+ID4gPiBJdCBkb2VzIEdVUCBhbmQgY3JlYXRlIEdFTSBvYmpl Y3QgQUZBSUNSIHlvdSBjYW4gd3JhcCB0aGF0IEdFTSBvYmplY3QgaW50byBhCj4gPiA+ID4gPiBk bWEgYnVmZmVyIG9iamVjdC4KPiA+ID4gPiA+IAo+ID4gPiA+IAo+ID4gPiA+IFRoYW5rIHlvdSBm b3IgZGlyZWN0aW5nIG1lIHRvIHRoaXMgaW1wbGVtZW50YXRpb24uIEl0IGlzIGludGVyZXN0aW5n OikuCj4gPiA+ID4gCj4gPiA+ID4gQnV0IGl0IGlzIG5vdCB5ZXQgc29sdmUgbXkgcHJvYmxlbS4g SWYgSSB1bmRlcnN0YW5kIGl0IHJpZ2h0LCB0aGUgdXNlcnB0ciBpbgo+ID4gPiA+IGk5MTUgZG8g dGhlIGZvbGxvd2luZzoKPiA+ID4gPiAKPiA+ID4gPiAxLiBUaGUgdXNlciBwcm9jZXNzIHNldHMg YSB1c2VyIHBvaW50ZXIgd2l0aCBzaXplIHRvIHRoZSBrZXJuZWwgdmlhIGlvY3RsLgo+ID4gPiA+ IDIuIFRoZSBrZXJuZWwgd3JhcHMgaXQgYXMgYSBkbWEtYnVmIGFuZCBrZWVwcyB0aGUgcHJvY2Vz cydzIG1tIGZvciBmdXJ0aGVyCj4gPiA+ID4gICAgcmVmZXJlbmNlLgo+ID4gPiA+IDMuIFRoZSB1 c2VyIHBhZ2VzIGFyZSBhbGxvY2F0ZWQsIEdVUGVkIG9yIERNQSBtYXBwZWQgdG8gdGhlIGRldmlj ZS4gU28gdGhlIGRhdGEKPiA+ID4gPiAgICBjYW4gYmUgc2hhcmVkIGJldHdlZW4gdGhlIHVzZXIg c3BhY2UgYW5kIHRoZSBoYXJkd2FyZS4KPiA+ID4gPiAKPiA+ID4gPiBCdXQgbXkgc2NlbmFyaW8g aXM6IAo+ID4gPiA+IAo+ID4gPiA+IDEuIFRoZSB1c2VyIHByb2Nlc3MgaGFzIHNvbWUgZGF0YSBp biB0aGUgdXNlciBzcGFjZSwgcG9pbnRlZCBieSBhIHBvaW50ZXIsIHNheQo+ID4gPiA+ICAgIHB0 cjEuIEFuZCB3aXRoaW4gdGhlIG1lbW9yeSwgdGhlcmUgbWF5IGJlIHNvbWUgb3RoZXIgcG9pbnRl cnMsIGxldCdzIHNheSBvbmUKPiA+ID4gPiAgICBvZiB0aGVtIGlzIHB0cjIuCj4gPiA+ID4gMi4g Tm93IEkgbmVlZCB0byBhc3NpZ24gcHRyMSAqZGlyZWN0bHkqIHRvIHRoZSBoYXJkd2FyZSBNTUlP IHNwYWNlLiBBbmQgdGhlCj4gPiA+ID4gICAgaGFyZHdhcmUgbXVzdCByZWZlciBwdHIxIGFuZCBw dHIyICpkaXJlY3RseSogZm9yIGRhdGEuCj4gPiA+ID4gCj4gPiA+ID4gVXNlcnB0ciBsZXRzIHRo ZSBoYXJkd2FyZSBhbmQgcHJvY2VzcyBzaGFyZSB0aGUgc2FtZSBtZW1vcnkgc3BhY2UuIEJ1dCBJ IG5lZWQKPiA+ID4gPiB0aGVtIHRvIHNoYXJlIHRoZSBzYW1lICphZGRyZXNzIHNwYWNlKi4gU28g SU9NTVUgaXMgYSBNVVNUIGZvciBXYXJwRHJpdmUsCj4gPiA+ID4gTk9JT01NVSBtb2RlLCBhcyBK ZWFuIHNhaWQsIGlzIGp1c3QgZm9yIHZlcmlmeWluZyBzb21lIG9mIHRoZSBwcm9jZWR1cmUgaXMg T0suCj4gPiA+IAo+ID4gPiBTbyB0byBiZSAxMDAlIGNsZWFyIHNob3VsZCB3ZSBfaWdub3JlXyB0 aGUgbm9uIFNWQS9TVk0gY2FzZSA/Cj4gPiA+IElmIHNvIHRoZW4gd2FpdCBmb3IgbmVjZXNzYXJ5 IFNWQS9TVk0gdG8gbGFuZCBhbmQgZG8gd2FycCBkcml2ZQo+ID4gPiB3aXRob3V0IG5vbiBTVkEv U1ZNIHBhdGguCj4gPiA+IAo+ID4gCj4gPiBJIHRoaW5rIHdlIHNob3VsZCBjbGVhciB0aGUgY29u Y2VwdCBvZiBTVkEvU1ZNIGhlcmUuIEFzIG15IHVuZGVyc3RhbmRpbmcsIFNoYXJlCj4gPiBWaXJ0 dWFsIEFkZHJlc3MvTWVtb3J5IG1lYW5zOiBhbnkgdmlydHVhbCBhZGRyZXNzIGluIGEgcHJvY2Vz cyBjYW4gYmUgdXNlZCBieQo+ID4gZGV2aWNlIGF0IHRoZSBzYW1lIHRpbWUuIFRoaXMgcmVxdWly ZXMgSU9NTVUgZGV2aWNlIHRvIHN1cHBvcnQgUEFTSUQuIEFuZAo+ID4gb3B0aW9uYWxseSwgaXQg cmVxdWlyZXMgdGhlIGZlYXR1cmUgb2YgcGFnZS1mYXVsdC1mcm9tLWRldmljZS4KPiAKPiBZZXMg d2UgYWdyZWUgb24gd2hhdCBTVkEvU1ZNIGlzLiBUaGVyZSBpcyBhIG9uZSBnb3RjaGEgdGhvdWdo dCwgYWNjZXNzCj4gdG8gcmFuZ2UgdGhhdCBhcmUgTU1JTyBtYXAgaWUgQ1BVIHBhZ2UgdGFibGUg cG9pbnRpbmcgdG8gSU8gbWVtb3J5LCBJSVJDCj4gaXQgaXMgdW5kZWZpbmVkIHdoYXQgaGFwcGVu cyBvbiBzb21lIHBsYXRmb3JtIGZvciBhIGRldmljZSB0cnlpbmcgdG8KPiBhY2Nlc3MgdGhvc2Ug dXNpbmcgU1ZBL1NWTS4KPiAKPiAKPiA+IEJ1dCBiZWZvcmUgdGhlIGZlYXR1cmUgaXMgc2V0dGxl ZCBkb3duLCBJT01NVSBjYW4gYmUgdXNlZCBpbW1lZGlhdGVseSBpbiB0aGUKPiA+IGN1cnJlbnQg a2VybmVsLiBUaGF0IG1ha2UgaXQgcG9zc2libGUgdG8gYXNzaWduIE9ORSBwcm9jZXNzJ3Mgdmly dHVhbCBhZGRyZXNzZXMKPiA+IHRvIHRoZSBkZXZpY2UncyBJT01NVSBwYWdlIHRhYmxlIHdpdGgg R1VQLiBUaGlzIG1ha2UgV2FycERyaXZlIHdvcmsgd2VsbCBmb3Igb25lCj4gPiBwcm9jZXNzLgo+ IAo+IFVIID8gSG93ID8gWW91IHdhbnQgdG8gR1VQIF9ldmVyeV8gc2luZ2xlIHZhbGlkIGFkZHJl c3MgaW4gdGhlIHByb2Nlc3MKPiBhbmQgbWFwIGl0IHRvIHRoZSBkZXZpY2UgPyBIb3cgZG8geW91 IGhhbmRsZSBuZXcgdm1hLCBwYWdlIGJlaW5nIHJlcGxhY2UKPiAoZGVzcGl0ZSBHVVAgYmVjYXVz ZSBvZiB0aGluZ3MgdGhhdCB1dGltYXRlbHkgY2FsbHMgemFwIHB0ZSkgLi4uCj4gCj4gQWdhaW4g aGVyZSB5b3Ugc2FpZCB0aGF0IHRoZSBkZXZpY2UgbXVzdCBiZSBhYmxlIHRvIGFjY2VzcyBfYW55 XyB2YWxpZAo+IHBvaW50ZXIuIFdpdGggR1VQIHRoaXMgaXMgaW5zYW5lLgo+IAo+IFNvIGkgYW0g YXNzdW1pbmcgdGhpcyBpcyBub3Qgd2hhdCB5b3Ugd2FudCB0byBkbyB3aXRob3V0IFNWQS9TVk0g aWUgd2l0aAo+IEdVUCB5b3UgaGF2ZSBhIGRpZmZlcmVudCBwcm9ncmFtbWluZyBtb2RlbCwgb25l IGluIHdoaWNoIHRoZSB1c2Vyc3BhY2UKPiBtdXN0IGZpcnN0IGJpbmQgX3JhbmdlXyBvZiBtZW1v cnkgdG8gdGhlIGRldmljZSBhbmQgZ2V0IGEgRE1BIGFkZHJlc3MKPiBmb3IgdGhlIHJhbmdlLgo+ IAo+IEFnYWluLCBHVVAgcmFuZ2Ugb2YgcHJvY2VzcyBhZGRyZXNzIHNwYWNlIHRvIG1hcCBpdCB0 byBhIGRldmljZSBzbyB0aGF0Cj4gdXNlcnNwYWNlIGNhbiB1c2UgdGhlIGRldmljZSBvbiB0aGUg bWFwcGVkIHJhbmdlIGlzIHNvbWV0aGluZyB0aGF0IGRvCj4gZXhpc3QgaW4gdmFyaW91cyBwbGFj ZXMgaW4gdGhlIGtlcm5lbC4KPiAKClllcyBzYW1lIGFzIHlvdXIgZXhwZWN0YXRpb24sIGluIFdh cnBEcml2ZSwgd2UgdXNlIHRoZSBjb25jZXB0IG9mICJzaGFyaW5nIiB0bwpkbyBzby4gSWYgc29t ZSBtZW1vcnkgaXMgZ29pbmcgdG8gYmUgc2hhcmVkIGFtb25nIHByb2Nlc3MgYW5kIGRldmljZXMs IHdlIHVzZQp3ZF9zaGFyZV9tZW0ocXVldWUsIHB0ciwgc2l6ZSkgdG8gc2hhcmUgdGhvc2UgbWVt b3J5LiBXaGVuIHRoZSBxdWV1ZSBpcyB3b3JraW5nCmluIHRoaXMgbW9kZSwgdGhlIHBvaW50IGlz IHZhbGlkIGluIHRob3NlIG1lbW9yeSBzZWdtZW50cy4gVGhlIHdkX3NoYXJlX21lbSBjYWxsCnZm aW8gZG1hIG1hcCBzeXNjYWxsIHdoaWNoIHdpbGwgZG8gR1VQLiAKCklmIFNWQS9TVk0gaXMgZW5h YmxlZCwgdXNlciBzcGFjZSBjYW4gc2V0IFNIQVJFX0FMTCBmbGFncyB0byB0aGUgcXVldWUuIFRo ZW4Kd2Rfc2hhcmVfbWVtKCkgaXMgbm90IG5lY2Vzc2FyeS4KClRoaXMgaXMgcmVhbGx5IG5vdCBw b3B1bGFyIHdoZW4gd2Ugc3RhcnRlZCB0aGUgd29yayBvbiBXYXJwRHJpdmUuIFRoZSBHVVAKZG9j dW1lbnQgc2FpZCBpdCBzaG91bGQgYmUgcHV0IHdpdGhpbiB0aGUgc2NvcGUgb2YgbW1fc2VtIGlz IGxvY2tlZC4gQmVjYXVzZSBHVVAKc2ltcGx5IGluY3JlYXNlIHRoZSBwYWdlIHJlZmNvdW50LCBu b3Qga2VlcCB0aGUgbWFwcGluZyBiZXR3ZWVuIHRoZSBwYWdlIGFuZCB0aGUKdm1hLiBXZSBrZWVw IG91ciB3b3JrIHRvZ2V0aGVyIHdpdGggVkZJTyB0byBtYWtlIHN1cmUgdGhlIHByb2JsZW0gY2Fu IGJlIHNvbHZlZAppbiBvbmUgZGVhbC4KCkFuZCBub3cgd2UgaGF2ZSBHVVAtbG9uZ3Rlcm0gYW5k IG1hbnkgYWNjb3VudGluZyB3b3JrIGluIFZGSU8sIHdlIGRvbid0IHdhbnQgdG8KZG8gdGhhdCBh Z2Fpbi4KCj4gPiBOb3cgV2UgYXJlIHRhbGtpbmcgYWJvdXQgU1ZBIGFuZCBQQVNJRCwganVzdCB0 byBtYWtlIHN1cmUgV2FycERyaXZlIGNhbiBiZW5lZml0Cj4gPiBmcm9tIHRoZSBmZWF0dXJlIGlu IHRoZSBmdXR1cmUuIEl0IGRvc2Ugbm90IG1lYW5zIFdhcnBEcml2ZSBpcyB1c2VsZXNzIGJlZm9y ZQo+ID4gdGhhdC4gQW5kIGl0IHdvcmtzIGZvciBvdXIgWmlwIGFuZCBSU0EgYWNjZWxlcmF0b3Jz IGluIHBoeXNpY2FsIHdvcmxkLgo+IAo+IEp1c3Qgbm90IHdpdGggcmFuZG9tIHByb2Nlc3MgYWRk cmVzcyAuLi4KPiAKPiA+ID4gSWYgeW91IHN0aWxsIHdhbnQgbm9uIFNWQS9TVk0gcGF0aCB3aGF0 IHlvdSB3YW50IHRvIGRvIG9ubHkgd29ya3MKPiA+ID4gaWYgYm90aCBwdHIxIGFuZCBwdHIyIGFy ZSBpbiBhIHJhbmdlIHRoYXQgaXMgRE1BIG1hcHBlZCB0byB0aGUKPiA+ID4gZGV2aWNlIChtb3Jl b3ZlciB5b3UgbmVlZCBETUEgYWRkcmVzcyB0byBtYXRjaCBwcm9jZXNzIGFkZHJlc3MKPiA+ID4g d2hpY2ggaXMgbm90IGFuIGVhc3kgZmVhdCkuCj4gPiA+IAo+ID4gPiBOb3cgZXZlbiBpZiB5b3Ug b25seSB3YW50IFNWQS9TVk0sIGkgZG8gbm90IHNlZSB3aGF0IGlzIHRoZSBwb2ludAo+ID4gPiBv ZiBkb2luZyB0aGlzIGluc2lkZSBWRklPLiBBTUQgR1BVIGRyaXZlciBkb2VzIG5vdCBhbmQgdGhl cmUgd291bGQKPiA+ID4gYmUgbm8gYmVuZWZpdCBmb3IgdGhlbSB0byBiZSB0aGVyZS4gV2VsbCBh IEFNRCBWRklPIG1kZXYgZGV2aWNlCj4gPiA+IGRyaXZlciBmb3IgUUVNVSBndWVzdCBtaWdodCBi ZSB1c2VmdWwgYnV0IHRoZXkgaGF2ZSBTVklPIElJUkMuCj4gPiA+IAo+ID4gPiBGb3IgU1ZBL1NW TSB5b3VyIHVzYWdlIG1vZGVsIGlzOgo+ID4gPiAKPiA+ID4gU2V0dXA6Cj4gPiA+ICAgICAtIHVz ZXIgc3BhY2UgY3JlYXRlIGEgd2FycCBkcml2ZSBjb250ZXh0IGZvciB0aGUgcHJvY2Vzcwo+ID4g PiAgICAgLSB1c2VyIHNwYWNlIGNyZWF0ZSBhIGRldmljZSBzcGVjaWZpYyBjb250ZXh0IGZvciB0 aGUgcHJvY2Vzcwo+ID4gPiAgICAgLSB1c2VyIHNwYWNlIGNyZWF0ZSBhIHVzZXIgc3BhY2UgY29t bWFuZCBxdWV1ZSBmb3IgdGhlIGRldmljZQo+ID4gPiAgICAgLSB1c2VyIHNwYWNlIGJpbmQgY29t bWFuZCBxdWV1ZQo+ID4gPiAKPiA+ID4gICAgIEF0IHRoaXMgcG9pbnQgdGhlIGtlcm5lbCBkcml2 ZXIgaGFzIGJvdW5kIHRoZSBwcm9jZXNzIGFkZHJlc3MKPiA+ID4gICAgIHNwYWNlIHRvIHRoZSBk ZXZpY2Ugd2l0aCBhIGNvbW1hbmQgcXVldWUgYW5kIHVzZXJzcGFjZQo+ID4gPiAKPiA+ID4gVXNh Z2U6Cj4gPiA+ICAgICAtIHVzZXIgc3BhY2Ugc2NoZWR1bGUgd29yayBhbmQgY2FsbCBhcHByb3By aWF0ZSBmbHVzaC91cGRhdGUKPiA+ID4gICAgICAgaW9jdGwgZnJvbSB0aW1lIHRvIHRpbWUuIE1p Z2h0IGJlIG9wdGlvbmFsIGRlcGVuZHMgb24gdGhlCj4gPiA+ICAgICAgIGhhcmR3YXJlLCBidXQg cHJvYmFibHkgYSBnb29kIGlkZWEgdG8gZW5mb3JjZSBzbyB0aGF0IGtlcm5lbAo+ID4gPiAgICAg ICBjYW4gdW5iaW5kIHRoZSBjb21tYW5kIHF1ZXVlIHRvIGJpbmQgYW5vdGhlciBwcm9jZXNzIGNv bW1hbmQKPiA+ID4gICAgICAgcXVldWUuCj4gPiA+ICAgICAuLi4KPiA+ID4gCj4gPiA+IENsZWFu dXA6Cj4gPiA+ICAgICAtIHVzZXIgc3BhY2UgdW5iaW5kIGNvbW1hbmQgcXVldWUKPiA+ID4gICAg IC0gdXNlciBzcGFjZSBkZXN0cm95IGRldmljZSBzcGVjaWZpYyBjb250ZXh0Cj4gPiA+ICAgICAt IHVzZXIgc3BhY2UgZGVzdHJveSB3YXJwIGRyaXZlIGNvbnRleHQKPiA+ID4gICAgIEFsbCB0aGUg YWJvdmUgY2FuIGJlIGltcGxpY2l0IHdoZW4gY2xvc2luZyB0aGUgZGV2aWNlIGZpbGUuCj4gPiA+ IAo+ID4gPiBTbyBhZ2FpbiBpbiB0aGUgYWJvdmUgbW9kZWwgaSBkbyBub3Qgc2VlIGFueXdoZXJl IHNvbWV0aGluZyBmcm9tCj4gPiA+IFZGSU8gdGhhdCB3b3VsZCBiZW5lZml0IHRoaXMgbW9kZWwu Cj4gPiA+IAo+ID4gCj4gPiBMZXQgbWUgc2hvdyB5b3UgaG93IHRoZSBtb2RlbCB3aWxsIGJlIGlm IEkgdXNlIFZGSU86Cj4gPiAKPiA+IFNldHVwIChLZXJuZWwgcGFydCkKPiA+IAktIEtlcm5lbCBk cml2ZXIgZG8gZXZlcnkgYXMgdXN1YWwgdG8gc2VydmUgdGhlIG90aGVyIGZ1bmN0aW9uYWxpdHks IE5JQwo+ID4gCSAgY2FuIHN0aWxsIGJlIHJlZ2lzdGVyZWQgdG8gbmV0ZGV2LCBlbmNyeXB0b3Ig Y2FuIHN0aWxsIGJlIHJlZ2lzdGVyZWQKPiA+IAkgIHRvIGNyeXB0by4uLgo+ID4gCS0gQXQgdGhl IHNhbWUgdGltZSwgdGhlIGRyaXZlciBjYW4gZGV2b3RlIHNvbWUgb2YgaXRzIGhhcmR3YXJlIHJl c291cmNlCj4gPiAJICBhbmQgcmVnaXN0ZXIgdGhlbSBhcyBhIG1kZXYgY3JlYXRvciB0byB0aGUg VkZJTyBmcmFtZXdvcmsuIFRoaXMganVzdAo+ID4gCSAgbmVlZCBsaW1pdGVkIGNoYW5nZSB0byB0 aGUgVkZJTyB0eXBlMSBkcml2ZXIuCj4gCj4gSW4gdGhlIGFib3ZlIFZGSU8gZG9lcyBub3QgaGVs cCB5b3Ugb25lIGJpdCAuLi4geW91IGNhbiBkbyB0aGF0IHdpdGgKPiBhcyBtdWNoIGNvZGUgd2l0 aCBuZXcgY29tbW9uIGRldmljZSBhcyBmcm9udCBlbmQuCj4gCj4gPiBTZXR1cCAoVXNlciBzcGFj ZSkKPiA+IAktIFN5c3RlbSBhZG1pbmlzdHJhdG9yIGNyZWF0ZSBtZGV2IHZpYSB0aGUgbWRldiBj cmVhdG9yIGludGVyZmFjZS4KPiA+IAktIEZvbGxvd2luZyBWRklPIHNldHVwIHJvdXRpbmUsIHVz ZXIgc3BhY2Ugb3BlbiB0aGUgbWRldidzIGdyb3VwLCB0aGVyZSBpcwo+ID4gCSAgb25seSBvbmUg Z3JvdXAgZm9yIG9uZSBkZXZpY2UuCj4gPiAJLSBXaXRob3V0IFBBU0lEIHN1cHBvcnQsIHlvdSBk b24ndCBuZWVkIHRvIGRvIGFueXRoaW5nLiBXaXRoIFBBU0lELCBiaW5kCj4gPiAJICB0aGUgUEFT SUQgdG8gdGhlIGRldmljZSB2aWEgVkZJTyBpbnRlcmZhY2UuCj4gPiAJLSBHZXQgdGhlIGRldmlj ZSBmcm9tIHRoZSBncm91cCB2aWEgVkZJTyBpbnRlcmZhY2UgYW5kIG1tYXAgaXQgdGhlIHVzZXIK PiA+IAkgIHNwYWNlIGZvciBkZXZpY2UncyBNTUlPIGFjY2VzcyAoZm9yIHRoZSBxdWV1ZSkuCj4g PiAJLSBNYXAgd2hhdGV2ZXIgbWVtb3J5IHlvdSBuZWVkIHRvIHNoYXJlIHdpdGggdGhlIGRldmlj ZSB3aXRoIFZGSU8KPiA+IAkgIGludGVyZmFjZS4KPiA+IAktIChvcHQpIEFkZCBtb3JlIGRldmlj ZXMgaW50byB0aGUgY29udGFpbmVyIGlmIHlvdSB3YW50IHRvIHNoYXJlIHRoZQo+ID4gCSAgc2Ft ZSBhZGRyZXNzIHNwYWNlIHdpdGggdGhlbQo+IAo+IFNvIGFsbCBWRklPIGJ1eXMgeW91IGhlcmUg aXMgYm9pbGVyIHBsYXRlIGNvZGUgdGhhdCBkb2VzIGluc2VydF9wZm4oKQo+IHRvIGhhbmRsZSBN TUlPIG1hcHBpbmcuIFdoaWNoIGlzIGp1c3QgY291cGxlIGh1bmRyZWQgbGluZXMgb2YgYm9pbGVy Cj4gcGxhdGUgY29kZS4KPiAKCk5vLiBXaXRoIFZGSU8sIEkgZG9uJ3QgbmVlZCB0bzoKCjEuIEdV UCBhbmQgYWNjb3VudGluZyBmb3IgUkxJTUlUX01FTUxPQ0sKMi4gS2VlcCBhbGwgR1VQIHBhZ2Vz IGZvciByZWxlYXNpbmcgKFZGSU8gdXNlcyB0aGUgcmJfdHJlZSB0byBkbyBzbykKMi4gSGFuZGxl IHRoZSBQQVNJRCBvbiBTTU1VIChBUk0ncyBJT01NVSkgbXlzZWxmLgozLiBNdWx0aXBsZSBkZXZp Y2VzIG1lbmFnZW1lbnQgKFZGSU8gdXNlcyBjb250YWluZXIgdG8gbWFuYWdlIHRoaXMpCgpBbmQg ZXZlbiBhcyBhIGJvaWxlciBwbGF0ZSwgaXQgaXMgdmFsdWVhYmxlLCB0aGUgbWVtb3J5IHRoaW5n IGlzIHNlbnNpdGl2ZQppbnRlcmZhY2UgdG8gdXNlciBzcGFjZSwgaXQgY2FuIGVhc2lseSBiZWNv bWUgYSBzZWN1cml0eSBwcm9ibGVtLiBJZiBJIGNhbgphY2hpZXZlIG15IHRhcmdldCB3aXRoaW4g dGhlIHNjb3BlIG9mIFZGSU8sIHdoeSBub3Q/IEF0IGxlYXNlIGl0IGhhcyBiZWVuCnByb3ZlZCB0 byBiZSBzYWZlIGZvciB0aGUgdGltZSBiZWluZy4KCj4gPiAKPiA+IENsZWFudXA6Cj4gPiAJLSBV c2VyIHNwYWNlIGNsb3NlIHRoZSBncm91cCBmaWxlIGhhbmRsZXIKPiA+IAktIFRoZXJlIHdpbGwg YmUgYSBwcm9ibGVtIHRvIGxldCB0aGUgb3RoZXIgcHJvY2VzcyBrbm93IHRoZSBtZGV2IGlzCj4g PiAJICBmcmVlZCB0byBiZSB1c2VkIGFnYWluLiBNeSBSRkN2MSBjaG9vc2UgYSBmaWxlIGhhbmRs ZXIgc29sdXRpb24uIEFsZXgKPiA+IAkgIGRvc2Ugbm90IGxpa2UgaXQuIEJ1dCBpdCBpcyBub3Qg YSBiaWcgcHJvYmxlbS4gV2UgY2FuIGFsd2F5cyBoYXZlIGEKPiA+IAkgIHNjaGVkdWxlciBwcm9j ZXNzIHRvIG1hbmFnZSB0aGUgc3RhdGUgb2YgdGhlIG1kZXYgb3IgZXZlbiB3ZSBjYW4KPiA+IAkg IHN3aXRjaCBiYWNrIHRvIHRoZSBSRkN2MSBzb2x1dGlvbiB3aXRob3V0IHRvbyBtdWNoIGVmZm9y dCBpZiB3ZSBsaWtlCj4gPiAJICBpbiB0aGUgZnV0dXJlLgo+IAo+IElmIHlvdSB3ZXJlIG91dHNp ZGUgVkZJTyB5b3Ugd291bGQgaGF2ZSBtb3JlIGZyZWVkb20gb24gaG93IHRvIGRvIHRoYXQuCj4g Rm9yIGluc3RhbmNlIHByb2Nlc3Mgb3BlbmluZyB0aGUgZGV2aWNlIGZpbGUgY2FuIGJlIHBsYWNl ZCBvbiBxdWV1ZSBhbmQKPiBmaXJzdCBvbmUgaW4gdGhlIHF1ZXVlIGdldCB0byB1c2UgdGhlIGRl dmljZSB1bnRpbCBpdCBjbG9zZXMvcmVsZWFzZSB0aGUKPiBkZXZpY2UuIFRoZW4gbmV4dCBvbmUg aW4gcXVldWUgZ2V0IHRoZSBkZXZpY2UgLi4uCgpZZXMuIEkgZG8gbGlrZSB0aGUgZmlsZSBoYW5k bGUgc29sdXRpb24uIEJ1dCBJIGhvcGUgdGhlIHNvbHV0aW9uIGJlY29tZSBtYXR1cmUKYXMgc29v biBhcyBwb3NzaWJsZS4gTWFueSBvZiBvdXIgcHJvZHVjdHMsIGFuZCBhcyBJIGtub3cgaW5jbHVk ZSBzb21lIG9mIG91cgpwYXJ0bmVycywgYXJlIHdhaXRpbmcgZm9yIGEgbG9uZyB0ZXJtIHNvbHV0 aW9uIGFzIGRpcmVjdGlvbi4gSWYgSSByZWx5IG9uIHNvbWUKdW5tYXR1cmUgc29sdXRpb24sIHRo ZXkgbWF5IGNob29zZSBzb21lIGRldmlhdGVkLCBjdXN0b21pemVkIHNvbHV0aW9uLiBUaGF0IHdp bGwKYmUgbXVjaCBoYXJtZnVsLiBDb21wYXJlIHRvIHRoaXMsIHRoZSBmcmVlZG9tIGlzIG5vdCBz byBpbXBvcnRhbnQuLi4KCj4gCj4gPiBFeGNlcHQgZm9yIHRoZSBtaW5pbXVtIHVwZGF0ZSB0byB0 aGUgdHlwZTEgZHJpdmVyIGFuZCB1c2Ugc2RtZGV2IHRvIG1hbmFnZSB0aGUKPiA+IGludGVycnVw dCBzaGFyaW5nLCBJIGRvbid0IG5lZWQgYW55IGV4dHJhIGNvZGUgdG8gZ2FpbiB0aGUgYWRkcmVz cyBzaGFyaW5nCj4gPiBjYXBhYmlsaXR5LiBBbmQgdGhlIGNhcGFiaWxpdHkgd2lsbCBiZSBzdHJl bmd0aGVuIGFsb25nIHdpdGggdGhlIHVwZ3JhZGUgb2YgVkZJTy4KPiA+IAo+ID4gPiAKPiA+ID4g PiA+ID4gQW5kIEkgZG9uJ3QgdW5kZXJzdGFuZCB3aHkgSSBzaG91bGQgYXZvaWQgdG8gdXNlIFZG SU8/IEFzIEFsZXggc2FpZCwgVkZJTyBpcyB0aGUKPiA+ID4gPiA+ID4gdXNlciBkcml2ZXIgZnJh bWV3b3JrLiBBbmQgSSBuZWVkIGV4YWN0bHkgYSB1c2VyIGRyaXZlciBpbnRlcmZhY2UuIFdoeSBz aG91bGQgSQo+ID4gPiA+ID4gPiBpbnZlbnQgYW5vdGhlciB3aGVlbD8gSXQgaGFzIG1vc3Qgb2Yg c3R1ZmYgSSBuZWVkOgo+ID4gPiA+ID4gPiAKPiA+ID4gPiA+ID4gMS4gQ29ubmVjdGluZyBtdWx0 aXBsZSBkZXZpY2VzIHRvIHRoZSBzYW1lIGFwcGxpY2F0aW9uIHNwYWNlCj4gPiA+ID4gPiA+IDIu IFBpbm5pbmcgYW5kIERNQSBmcm9tIHRoZSBhcHBsaWNhdGlvbiBzcGFjZSB0byB0aGUgd2hvbGUg c2V0IG9mIGRldmljZQo+ID4gPiA+ID4gPiAzLiBNYW5hZ2luZyBoYXJkd2FyZSByZXNvdXJjZSBi eSBkZXZpY2UKPiA+ID4gPiA+ID4gCj4gPiA+ID4gPiA+IFdlIGp1c3QgbmVlZCB0aGUgbGFzdCBz dGVwOiBtYWtlIHN1cmUgbXVsdGlwbGUgYXBwbGljYXRpb25zIGFuZCB0aGUga2VybmVsIGNhbgo+ ID4gPiA+ID4gPiBzaGFyZSB0aGUgc2FtZSBJT01NVS4gVGhlbiB3aHkgc2hvdWxkbid0IHdlIHVz ZSBWRklPPwo+ID4gPiA+ID4gCj4gPiA+ID4gPiBCZWNhdXNlIHRvbnMgb2Ygb3RoZXIgZHJpdmVy cyBhbHJlYWR5IGRvIGFsbCBvZiB0aGUgYWJvdmUgb3V0c2lkZSBWRklPLiBNYW55Cj4gPiA+ID4g PiBkcml2ZXIgaGF2ZSBhIHNpemVhYmxlIHVzZXJzcGFjZSBzaWRlIHRvIHRoZW0gKGFueXRoaW5n IHdpdGggaW9jdGwgZG8pIHNvIHRoZXkKPiA+ID4gPiA+IGNhbiBiZSBjb25zdHJ1ZGVkIGFzIHVz ZXJzcGFjZSBkcml2ZXIgdG9vLgo+ID4gPiA+ID4gCj4gPiA+ID4gCj4gPiA+ID4gSWdub3Jpbmcg aWYgdGhlcmUgYXJlICp0b25zKiBvZiBkcml2ZXJzIGFyZSBkb2luZyB0aGF0OyksIGV2ZW4gSSBk byB0aGUgc2FtZSBhcwo+ID4gPiA+IGk5MTUgYW5kIHNvbHZlIHRoZSBhZGRyZXNzIHNwYWNlIHBy b2JsZW0uIEFuZCBpZiBJIGRvbid0IG5lZWQgdG8gd2l0aCBWRklPLCB3aHkKPiA+ID4gPiBzaG91 bGQgSSBzcGVuZCBzbyBtdWNoIGVmZm9ydCB0byBkbyBpdCBhZ2Fpbj8KPiA+ID4gCj4gPiA+IEJl Y2F1c2UgeW91IGRvIG5vdCBuZWVkIGFueSBjb2RlIGZyb20gVkZJTywgbm9yIGRvIHlvdSBuZWVk IHRvIHJlaW52ZW50Cj4gPiA+IHRoaW5ncy4gSWYgbm9uIFNWQS9TVk0gbWF0dGVycyB0byB5b3Ug dGhlbiB1c2UgZG1hIGJ1ZmZlci4gSWYgbm90IHRoZW4KPiA+ID4gaSBkbyBub3Qgc2VlIGFueXRo aW5nIGluIFZGSU8gdGhhdCB5b3UgbmVlZC4KPiA+ID4gCj4gPiAKPiA+IEFzIEkgaGF2ZSBleHBs YWluLCBpZiBJIGRvbid0IHVzZSBWRklPLCBhdCBsZWFzZSBJIGhhdmUgdG8gZG8gYWxsIHRoYXQg aGFzIGJlZW4KPiA+IGRvbmUgaW4gaTkxNSBvciBldmVuIG1vcmUgdGhhbiB0aGF0Lgo+IAo+IFNv IGJlc2lkZSB0aGUgTU1JTyBtbWFwKCkgaGFuZGxpbmcgYW5kIGRtYSBtYXBwaW5nIG9mIHJhbmdl IG9mIHVzZXIgc3BhY2UKPiBhZGRyZXNzIHNwYWNlIChhZ2FpbiBhbGwgdmVyeSBib2lsZXIgcGxh dGUgY29kZSBkdXBsaWNhdGVkIGFjY3Jvc3MgdGhlCj4ga2VybmVsIHNldmVyYWwgdGltZSBpbiBk aWZmZXJlbnQgZm9ybXMpLiBZb3UgZG8gbm90IGdhaW4gYW55dGhpbmcgYmVpbmcKPiBpbnNpZGUg VkZJTyByaWdodCA/Cj4gCgpBcyBJIHNhaWQsIHJiLXRyZWUgZm9yIGd1cCwgcmxpbWl0IGFjY291 bnRpbmcsIGNvb3BlcmF0aW9uIG9uIFNNTVUsIGFuZCBtYXR1cmUKdXNlciBpbnRlcmZhY2UgYXJl IG91ciBjb25jZXJuLgoKPiAKPiA+ID4gPiA+IFNvIHRoZXJlIGlzIG5vIHJlYXNvbnMgdG8gZG8g dGhhdCB1bmRlciBWRklPLiBFc3BlY2lhbHkgYXMgaW4geW91ciBleGFtcGxlCj4gPiA+ID4gPiBp dCBpcyBub3QgYSByZWFsIHVzZXIgc3BhY2UgZGV2aWNlIGRyaXZlciwgdGhlIHVzZXJzcGFjZSBw b3J0aW9uIG9ubHkga25vd3MKPiA+ID4gPiA+IGFib3V0IHdyaXR0aW5nIGNvbW1hbmQgaW50byBj b21tYW5kIGJ1ZmZlciBBRkFJQ1QuCj4gPiA+ID4gPiAKPiA+ID4gPiA+IFZGSU8gaXMgZm9yIHJl YWwgdXNlcnNwYWNlIGRyaXZlciB3aGVyZSBpbnRlcnJ1cHQsIGNvbmZpZ3VyYXRpb25zLCAuLi4g aWUKPiA+ID4gPiA+IGFsbCB0aGUgZHJpdmVyIGlzIGhhbmRsZWQgaW4gdXNlcnNwYWNlLiBUaGlz IG1lYW5zIHRoYXQgdGhlIHVzZXJzcGFjZSBoYXZlCj4gPiA+ID4gPiB0byBiZSB0cnVzdGVkIGFz IGl0IGNvdWxkIHByb2dyYW0gdGhlIGRldmljZSB0byBkbyBETUEgdG8gYW55d2hlcmUgKGlmCj4g PiA+ID4gPiBJT01NVSBpcyBkaXNhYmxlZCBhdCBib290IHdoaWNoIGlzIHN0aWxsIHRoZSBkZWZh dWx0IGNvbmZpZ3VyYXRpb24gaW4gdGhlCj4gPiA+ID4gPiBrZXJuZWwpLgo+ID4gPiA+ID4gCj4g PiA+ID4gCj4gPiA+ID4gQnV0IGFzIEFsZXggZXhwbGFpbmVkLCBWRklPIGlzIG5vdCBzaW1wbHkg dXNlZCBieSBWTS4gU28gaXQgbmVlZCBub3QgdG8gaGF2ZSBhbGwKPiA+ID4gPiBzdHVmZnMgYXMg YSBkcml2ZXIgaW4gaG9zdCBzeXN0ZW0uIEFuZCBJIGRvIG5lZWQgdG8gc2hhcmUgdGhlIHVzZXIg c3BhY2UgYXMgRE1BCj4gPiA+ID4gYnVmZmVyIHRvIHRoZSBoYXJkd2FyZS4gQW5kIEkgY2FuIGdl dCBpdCB3aXRoIGp1c3QgYSBsaXR0bGUgdXBkYXRlLCB0aGVuIGl0IGNhbgo+ID4gPiA+IHNlcnZp Y2UgbWUgcGVyZmVjdGx5LiBJIGRvbid0IHVuZGVyc3RhbmQgd2h5IEkgc2hvdWxkIGNob29zZSBh IGxvbmcgcm91dGUuCj4gPiA+IAo+ID4gPiBBZ2FpbiB0aGlzIGlzIG5vdCB0aGUgbG9uZyByb3V0 ZSBpIGRvIG5vdCBzZWUgYW55dGhpbmcgaW4gVkZJTyB0aGF0Cj4gPiA+IGJlbmVmaXQgeW91IGlu IHRoZSBTVkEvU1ZNIGNhc2UuIEEgYmFzaWMgY2hhcmFjdGVyIGRldmljZSBkcml2ZXIgY2FuCj4g PiA+IGRvIHRoYXQuCj4gPiA+IAo+ID4gPiAKPiA+ID4gPiA+IFNvIGkgZG8gbm90IHNlZSBhbnkg cmVhc29ucyB0byBkbyBhbnl0aGluZyB5b3Ugd2FudCBpbnNpZGUgVkZJTy4gQWxsIHlvdQo+ID4g PiA+ID4gd2FudCB0byBkbyBjYW4gYmUgZG9uZSBvdXRzaWRlIGFzIGVhc2lseS4gTW9yZW92ZXIg aXQgd291bGQgYmUgYmV0dGVyIGlmCj4gPiA+ID4gPiB5b3UgZGVmaW5lIGNsZWFybHkgZWFjaCBz Y2VuYXJpbyBiZWNhdXNlIGZyb20gd2hlcmUgaSBzaXQgaXQgbG9va3MgbGlrZQo+ID4gPiA+ID4g eW91IGFyZSBvcGVuaW5nIHRoZSBkb29yIHdpZGUgb3BlbiB0byB1c2Vyc3BhY2UgdG8gRE1BIGFu eXdoZXJlIHdoZW4gSU9NTVUKPiA+ID4gPiA+IGlzIGRpc2FibGVkLgo+ID4gPiA+ID4gCj4gPiA+ ID4gPiBXaGVuIElPTU1VIGlzIGRpc2FibGVkIHlvdSBjYW4gX25vdF8gZXhwb3NlIGNvbW1hbmQg cXVldWUgdG8gdXNlcnNwYWNlCj4gPiA+ID4gPiB1bmxlc3MgeW91ciBkZXZpY2UgaGFzIGl0cyBv d24gcGFnZSB0YWJsZSBhbmQgYWxsIGNvbW1hbmRzIGFyZSByZWxhdGl2ZQo+ID4gPiA+ID4gdG8g dGhhdCBwYWdlIHRhYmxlIGFuZCB0aGUgZGV2aWNlIHBhZ2UgdGFibGUgaXMgcG9wdWxhdGVkIGJ5 IGtlcm5lbCBkcml2ZXIKPiA+ID4gPiA+IGluIHNlY3VyZSB3YXkgKGllIGJ5IGNoZWNraW5nIHRo YXQgd2hhdCBpcyBwb3B1bGF0ZWQgY2FuIGJlIGFjY2VzcykuCj4gPiA+ID4gPiAKPiA+ID4gPiA+ IEkgZG8gbm90IGJlbGlldmUgeW91ciBleGFtcGxlIGRldmljZSB0byBoYXZlIHN1Y2ggcGFnZSB0 YWJsZSBub3IgZG8gaSBzZWUKPiA+ID4gPiA+IGEgZmFsbGJhY2sgcGF0aCB3aGVuIElPTU1VIGlz IGRpc2FibGVkIHRoYXQgZm9yY2UgdXNlciB0byBkbyBpb2N0bCBmb3IKPiA+ID4gPiA+IGVhY2gg Y29tbWFuZHMuCj4gPiA+ID4gPiAKPiA+ID4gPiA+IFllcyBpIHVuZGVyc3RhbmQgdGhhdCB5b3Ug dGFyZ2V0IFNWQS9TVk0gYnV0IHN0aWxsIHlvdSBjbGFpbSB0byBzdXBwb3J0Cj4gPiA+ID4gPiBu b24gU1ZBL1NWTS4gVGhlIHBvaW50IGlzIHRoYXQgdXNlcnNwYWNlIGNhbiBub3QgYmUgdHJ1c3Rl ZCBpZiB5b3Ugd2FudAo+ID4gPiA+ID4gdG8gaGF2ZSByYW5kb20gcHJvZ3JhbSB1c2UgeW91ciBk ZXZpY2UuIEkgYW0gcHJldHR5IHN1cmUgdGhhdCBhbGwgdXNlcgo+ID4gPiA+ID4gb2YgVkZJTyBh cmUgdHJ1c3RlZCBwcm9jZXNzIChsaWtlIFFFTVUpLgo+ID4gPiA+ID4gCj4gPiA+ID4gPiAKPiA+ ID4gPiA+IEZpbmFseSBpIGFtIGNvbnZpbmNlIHRoYXQgdGhlIElPTU1VIGdyb3VwaW5nIHN0dWZm IHJlbGF0ZWQgdG8gVkZJTyBpcwo+ID4gPiA+ID4gdXNlbGVzcyBmb3IgeW91ciB1c2VjYXNlLiBJ IHJlYWxseSBkbyBub3Qgc2VlIHRoZSBwb2ludCBvZiB0aGF0LCBpdAo+ID4gPiA+ID4gZG9lcyBj b21wbGljYXRlIHRoaW5ncyBmb3IgeW91IGZvciBubyByZWFzb25zIEFGQUlDVC4KPiA+ID4gPiAK PiA+ID4gPiBJbmRlZWQsIEkgZG9uJ3QgbGlrZSB0aGUgZ3JvdXAgdGhpbmcuIEkgYmVsaWV2ZSBW RklPJ3MgbWFpbnRhaW5zIHdvdWxkIG5vdCBsaWtlCj4gPiA+ID4gaXQgdmVyeSBtdWNoIGVpdGhl cjspLiBCdXQgdGhlIHByb2JsZW0gaXMsIHRoZSBncm91cCByZWZsZWN0cyB0byB0aGUgc2FtZQo+ ID4gPiA+IElPTU1VKHVuaXQpLCB3aGljaCBtYXkgc2hhcmVkIHdpdGggb3RoZXIgZGV2aWNlcy4g IEl0IGlzIGEgc2VjdXJpdHkgcHJvYmxlbS4gSQo+ID4gPiA+IGNhbm5vdCBpZ25vcmUgaXQuIEkg aGF2ZSB0byB0YWtlIGl0IGludG8gYWNjb3VudCBldmVudCBJIGRvbid0IHVzZSBWRklPLgo+ID4g PiAKPiA+ID4gVG8gbWUgaXQgc2VlbXMgeW91IGFyZSBtYWtpbmcgYSBwb2xpY3kgZGVjaXNzaW9u IGluIGtlcm5lbCBzcGFjZSBpZQo+ID4gPiB3ZXRoZXIgdGhlIGRldmljZSBzaG91bGQgYmUgaXNv bGF0ZWQgaW4gaXRzIG93biBncm91cCBvciBub3QgaXMgYQo+ID4gPiBkZWNpc3Npb24gdGhhdCBp cyB1cCB0byB0aGUgc3lzIGFkbWluIG9yIHNvbWV0aGluZyBpbiB1c2Vyc3BhY2UuCj4gPiA+IFJp Z2h0IG5vdyBleGlzdGluZyB1c2VyIG9mIFNWQS9TVk0gZG9uJ3QgKGF0IGxlYXN0IEFGQUlDVCku Cj4gPiA+IAo+ID4gPiBEbyB3ZSByZWFsbHkgd2FudCB0byBmb3JjZSBzdWNoIGlzb2xhdGlvbiA/ Cj4gPiA+IAo+ID4gCj4gPiBCdXQgaXQgaXMgbm90IG15IGRlY2lzaW9uLCB0aGF0IGhvdyB0aGUg aW9tbXUgc3Vic3lzdGVtIGlzIGRlc2lnbmVkLiBQZXJzb25hbGx5Cj4gPiBJIGRvbid0IGxpa2Ug aXQgYXQgYWxsLCBiZWNhdXNlIGFsbCBvdXIgaGFyZHdhcmVzIGhhdmUgdGhlaXIgb3duIHN0cmVh bSBpZAo+ID4gKGRldmljZSBpZCkuIEkgZG9uJ3QgbmVlZCB0aGUgZ3JvdXAgY29uY2VwdCBhdCBh bGwuIEJ1dCB0aGUgaW9tbXUgc3Vic3lzdGVtCj4gPiBhc3N1bWUgc29tZSBkZXZpY2VzIG1heSBz aGFyZSB0aGUgbmFtZSBkZXZpY2UgSUQgdG8gYSBzaW5nbGUgSU9NTVUuCj4gCj4gTXkgcXVlc3Rp b24gd2FzIGRvIHlvdSByZWFsbHkgd2FudCB0byBmb3JjZSBncm91cCBpc29sYXRpb24gZm9yIHRo ZQo+IGRldmljZSA/IEV4aXN0aW5nIFNWQS9TVk0gY2FwYWJsZSBkcml2ZXIgZG8gbm90IGZvcmNl IHRoYXQsIHRoZXkgbGV0Cj4gdGhlIHVzZXJzcGFjZSBkZWNpZGUgdGhpcyAoc3lzYWRtLCBkaXN0 cmlidXRpb25zLCAuLi4pLiBCZWluZyBwYXJ0IG9mCj4gVkZJTyAoaW4gdGhlIHdheSB5b3UgZG8s IGxpa2VseSB3YXlzIHRvIGF2b2lkIHRoaXMgaW5zaWRlIFZGSU8gdG9vKQo+IGZvcmNlIHRoaXMg ZGVjaXNpb24gaWUgbWFrZSBhIHBvbGljeSBkZWNpc2lvbiB3aXRob3V0IHVzZXJzcGFjZSBoYXZp bmcKPiBhbnl0aGluZyB0byBzYXkgYWJvdXQgaXQuCj4gCj4gCj4gVGhlIElPTU1VIGdyb3VwIHRo aW5nIGFzIGFsd2F5cyBiZWVuIGRvdWJ0IGZ1bGwgdG8gbWUsIGl0IGlzIGFkdmVydGlzZQo+IGFz IGFsbG93aW5nIHRvIHNoYXJlIHJlc291cmNlcyAoaWUgSU9NTVUgcGFnZSB0YWJsZSkgYmV0d2Vl biBkZXZpY2VzLgo+IEJ1dCB0aGlzIGFzc3VtZSB0aGF0IGFsbCBkZXZpY2UgZHJpdmVyIGluIHRo ZSBncm91cCBoYXZlIHNvbWUgd2F5IG9mCj4gY29tbXVuaWNhdGluZyB3aXRoIGVhY2ggb3RoZXIg dG8gc2hhcmUgY29tbW9uIERNQSBhZGRyZXNzIHRoYXQgcG9pbnQKPiB0byBtZW1vcnkgZGV2aWNl cyBjYXJlLiBJIGJlbGlldmUgb25seSBWRklPIGRvZXMgdGhhdCBhbmQgcHJvYmFibHkKPiBvbmx5 IHdoZW4gdXNlIGJ5IFFFTVUuCj4gCj4gCj4gQW55d2F5IG15IHF1ZXN0aW9uIGlzOgo+IAo+IElz IGl0IHRoYXQgbXVjaCB1c2VmdWwgdG8gYmUgaW5zaWRlIFZGSU8gKHRvIGF2b2lkIGZldyBodW5k cmVkIGxpbmVzCj4gb2YgYm9pbGVyIHBsYXRlIGNvZGUpIGdpdmVuIHRoYXQgaXQgZm9yY2VzIHlv dSBpbnRvIGEgbW9kZWwgKGdyb3VwCj4gaXNvbGF0aW9uKSB0aGF0IHNvIGZhciBoYXZlIG5ldmVy IGJlZW4gdGhlIHByZWZlcmVkIHdheSBmb3IgYWxsCj4gZXhpc3RpbmcgZGV2aWNlIGRyaXZlciB0 aGF0IGFscmVhZHkgZG8gd2hhdCB5b3Ugd2FudCB0byBhY2hpZXZlID8KPiAKCllvdSBtZWFuIHRv IHNheSBJIGNyZWF0ZSBhbm90aGVyIGZyYW1ld29yayBhbmQgY29weSBtb3N0IG9mIHRoZSBjb2Rl IGZyb20gVkZJTz8KSXQgaXMgaGFyZCB0byBiZWxpZXZlIHRoZSBtYWlubGluZSBrZXJuZWwgd2ls bCB0YWtlIG15IGNvZGUuIFNvIGhvdyBhYm91dCBsZXQgbWUKdHJ5IHRoZSBWRklPIHdheSBmaXJz dCBhbmQgdHJ5IHRoYXQgaWYgaXQgd29uJ3Qgd29yaz8gOykKCj4gCj4gPkZyb20gd2hlcmUgaSBz dGFuZCBpIGRvIG5vdCBzZWUgb3ZlcndoZWxtaW5nIHJlYXNvbnMgdG8gZG8gd2hhdCB5b3UKPiBh cmUgZG9pbmcgaW5zaWRlIFZGSU8uCj4gCj4gVG8gbWUgaXQgd291bGQgbWFrZSBtb3JlIHNlbnNl IHRvIGhhdmUgcmVndWxhciBkZXZpY2UgZHJpdmVyLiBUaGV5Cj4gYWxsIGNhbiBoYXZlIGRldmlj ZSBmaWxlIHVuZGVyIHNhbWUgaGllcmFyY2h5IHRvIG1ha2UgZGV2aWNlcyB3aXRoCj4gc2FtZSBw cm9ncmFtbWluZyBtb2RlbCBlYXN5IHRvIGRpc2NvdmVyLgo+IAo+IErDqXLDtG1lCgpDaGVlcnMK LS0gCgkJCS1LZW5uZXRoKEhpc2lsaWNvbikKCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09CuacrOmC ruS7tuWPiuWFtumZhOS7tuWQq+acieWNjuS4uuWFrOWPuOeahOS/neWvhuS/oeaBr++8jOS7hemZ kOS6juWPkemAgee7meS4iumdouWcsOWdgOS4reWIl+WHuueahOS4quS6uuaIlue+pOe7hOOAguem gQrmraLku7vkvZXlhbbku5bkurrku6Xku7vkvZXlvaLlvI/kvb/nlKjvvIjljIXmi6zkvYbkuI3p mZDkuo7lhajpg6jmiJbpg6jliIblnLDms4TpnLLjgIHlpI3liLbjgIHmiJbmlaPlj5HvvInmnKzp gq7ku7bkuK0K55qE5L+h5oGv44CC5aaC5p6c5oKo6ZSZ5pS25LqG5pys6YKu5Lu277yM6K+35oKo 56uL5Y2z55S16K+d5oiW6YKu5Lu26YCa55+l5Y+R5Lu25Lq65bm25Yig6Zmk5pys6YKu5Lu277yB ClRoaXMgZS1tYWlsIGFuZCBpdHMgYXR0YWNobWVudHMgY29udGFpbiBjb25maWRlbnRpYWwgaW5m b3JtYXRpb24gZnJvbSBIVUFXRUksCndoaWNoIGlzIGludGVuZGVkIG9ubHkgZm9yIHRoZSBwZXJz b24gb3IgZW50aXR5IHdob3NlIGFkZHJlc3MgaXMgbGlzdGVkIGFib3ZlLgpBbnkgdXNlIG9mIHRo ZSAKaW5mb3JtYXRpb24gY29udGFpbmVkIGhlcmVpbiBpbiBhbnkgd2F5IChpbmNsdWRpbmcsIGJ1 dCBub3QgbGltaXRlZCB0bywgdG90YWwgb3IKcGFydGlhbCBkaXNjbG9zdXJlLCByZXByb2R1Y3Rp b24sIG9yIGRpc3NlbWluYXRpb24pIGJ5IHBlcnNvbnMgb3RoZXIgdGhhbiB0aGUKaW50ZW5kZWQg CnJlY2lwaWVudChzKSBpcyBwcm9oaWJpdGVkLiBJZiB5b3UgcmVjZWl2ZSB0aGlzIGUtbWFpbCBp biBlcnJvciwgcGxlYXNlIG5vdGlmeQp0aGUgc2VuZGVyIGJ5IHBob25lIG9yIGVtYWlsIGltbWVk aWF0ZWx5IGFuZCBkZWxldGUgaXQhCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwppb21tdSBtYWlsaW5nIGxpc3QKaW9tbXVAbGlzdHMubGludXgtZm91bmRh dGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGlu Zm8vaW9tbXU= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3603BC04ABB for ; Tue, 11 Sep 2018 06:42:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C1E2A2086A for ; Tue, 11 Sep 2018 06:42:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C1E2A2086A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=hisilicon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727664AbeIKLkg (ORCPT ); Tue, 11 Sep 2018 07:40:36 -0400 Received: from szxga06-in.huawei.com ([45.249.212.32]:51766 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726481AbeIKLkg (ORCPT ); Tue, 11 Sep 2018 07:40:36 -0400 Received: from DGGEMS413-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 11A061347CBB0; Tue, 11 Sep 2018 14:42:43 +0800 (CST) Received: from localhost (10.67.212.75) by DGGEMS413-HUB.china.huawei.com (10.3.19.213) with Microsoft SMTP Server (TLS) id 14.3.399.0; Tue, 11 Sep 2018 14:42:34 +0800 Date: Tue, 11 Sep 2018 14:40:43 +0800 From: Kenneth Lee To: Jerome Glisse CC: Kenneth Lee , Zaibo Xu , "Herbert Xu" , , Jonathan Corbet , Greg Kroah-Hartman , "Joerg Roedel" , , Sanjay Kumar , Hao Fang , , , , Alex Williamson , , Zhou Wang , "Philippe Ombredanne" , Thomas Gleixner , "David S . Miller" , , Lu Baolu Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive Message-ID: <20180911064043.GA207969@Turing-Arch-b> References: <20180904150019.GA4024@redhat.com> <20180904101509.62314b67@t450s.home> <20180906094532.GG230707@Turing-Arch-b> <20180906133133.GA3830@redhat.com> <20180907040138.GI230707@Turing-Arch-b> <20180907165303.GA3519@redhat.com> <20180910032809.GJ230707@Turing-Arch-b> <20180910145423.GA3488@redhat.com> <20180911024209.GK230707@Turing-Arch-b> <20180911033358.GA4730@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180911033358.GA4730@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Originating-IP: [10.67.212.75] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Sep 10, 2018 at 11:33:59PM -0400, Jerome Glisse wrote: > Date: Mon, 10 Sep 2018 23:33:59 -0400 > From: Jerome Glisse > To: Kenneth Lee > CC: Kenneth Lee , Zaibo Xu , > Herbert Xu , kvm@vger.kernel.org, Jonathan > Corbet , Greg Kroah-Hartman , > Joerg Roedel , linux-doc@vger.kernel.org, Sanjay Kumar > , Hao Fang , > iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, > linuxarm@huawei.com, Alex Williamson , > linux-crypto@vger.kernel.org, Zhou Wang , > Philippe Ombredanne , Thomas Gleixner > , "David S . Miller" , > linux-accelerators@lists.ozlabs.org, Lu Baolu > Subject: Re: [RFCv2 PATCH 0/7] A General Accelerator Framework, WarpDrive > User-Agent: Mutt/1.10.1 (2018-07-13) > Message-ID: <20180911033358.GA4730@redhat.com> > > On Tue, Sep 11, 2018 at 10:42:09AM +0800, Kenneth Lee wrote: > > On Mon, Sep 10, 2018 at 10:54:23AM -0400, Jerome Glisse wrote: > > > On Mon, Sep 10, 2018 at 11:28:09AM +0800, Kenneth Lee wrote: > > > > On Fri, Sep 07, 2018 at 12:53:06PM -0400, Jerome Glisse wrote: > > > > > On Fri, Sep 07, 2018 at 12:01:38PM +0800, Kenneth Lee wrote: > > > > > > On Thu, Sep 06, 2018 at 09:31:33AM -0400, Jerome Glisse wrote: > > > > > > > On Thu, Sep 06, 2018 at 05:45:32PM +0800, Kenneth Lee wrote: > > > > > > > > On Tue, Sep 04, 2018 at 10:15:09AM -0600, Alex Williamson wrote: > > > > > > > > > On Tue, 4 Sep 2018 11:00:19 -0400 Jerome Glisse wrote: > > > > > > > > > > On Mon, Sep 03, 2018 at 08:51:57AM +0800, Kenneth Lee wrote: > > > > > > [...] > > > > > > > > > I took a look at i915_gem_execbuffer_ioctl(). It seems it "copy_from_user" the > > > > > > user memory to the kernel. That is not what we need. What we try to get is: the > > > > > > user application do something on its data, and push it away to the accelerator, > > > > > > and says: "I'm tied, it is your turn to do the job...". Then the accelerator has > > > > > > the memory, referring any portion of it with the same VAs of the application, > > > > > > even the VAs are stored inside the memory itself. > > > > > > > > > > You were not looking at right place see drivers/gpu/drm/i915/i915_gem_userptr.c > > > > > It does GUP and create GEM object AFAICR you can wrap that GEM object into a > > > > > dma buffer object. > > > > > > > > > > > > > Thank you for directing me to this implementation. It is interesting:). > > > > > > > > But it is not yet solve my problem. If I understand it right, the userptr in > > > > i915 do the following: > > > > > > > > 1. The user process sets a user pointer with size to the kernel via ioctl. > > > > 2. The kernel wraps it as a dma-buf and keeps the process's mm for further > > > > reference. > > > > 3. The user pages are allocated, GUPed or DMA mapped to the device. So the data > > > > can be shared between the user space and the hardware. > > > > > > > > But my scenario is: > > > > > > > > 1. The user process has some data in the user space, pointed by a pointer, say > > > > ptr1. And within the memory, there may be some other pointers, let's say one > > > > of them is ptr2. > > > > 2. Now I need to assign ptr1 *directly* to the hardware MMIO space. And the > > > > hardware must refer ptr1 and ptr2 *directly* for data. > > > > > > > > Userptr lets the hardware and process share the same memory space. But I need > > > > them to share the same *address space*. So IOMMU is a MUST for WarpDrive, > > > > NOIOMMU mode, as Jean said, is just for verifying some of the procedure is OK. > > > > > > So to be 100% clear should we _ignore_ the non SVA/SVM case ? > > > If so then wait for necessary SVA/SVM to land and do warp drive > > > without non SVA/SVM path. > > > > > > > I think we should clear the concept of SVA/SVM here. As my understanding, Share > > Virtual Address/Memory means: any virtual address in a process can be used by > > device at the same time. This requires IOMMU device to support PASID. And > > optionally, it requires the feature of page-fault-from-device. > > Yes we agree on what SVA/SVM is. There is a one gotcha thought, access > to range that are MMIO map ie CPU page table pointing to IO memory, IIRC > it is undefined what happens on some platform for a device trying to > access those using SVA/SVM. > > > > But before the feature is settled down, IOMMU can be used immediately in the > > current kernel. That make it possible to assign ONE process's virtual addresses > > to the device's IOMMU page table with GUP. This make WarpDrive work well for one > > process. > > UH ? How ? You want to GUP _every_ single valid address in the process > and map it to the device ? How do you handle new vma, page being replace > (despite GUP because of things that utimately calls zap pte) ... > > Again here you said that the device must be able to access _any_ valid > pointer. With GUP this is insane. > > So i am assuming this is not what you want to do without SVA/SVM ie with > GUP you have a different programming model, one in which the userspace > must first bind _range_ of memory to the device and get a DMA address > for the range. > > Again, GUP range of process address space to map it to a device so that > userspace can use the device on the mapped range is something that do > exist in various places in the kernel. > Yes same as your expectation, in WarpDrive, we use the concept of "sharing" to do so. If some memory is going to be shared among process and devices, we use wd_share_mem(queue, ptr, size) to share those memory. When the queue is working in this mode, the point is valid in those memory segments. The wd_share_mem call vfio dma map syscall which will do GUP. If SVA/SVM is enabled, user space can set SHARE_ALL flags to the queue. Then wd_share_mem() is not necessary. This is really not popular when we started the work on WarpDrive. The GUP document said it should be put within the scope of mm_sem is locked. Because GUP simply increase the page refcount, not keep the mapping between the page and the vma. We keep our work together with VFIO to make sure the problem can be solved in one deal. And now we have GUP-longterm and many accounting work in VFIO, we don't want to do that again. > > Now We are talking about SVA and PASID, just to make sure WarpDrive can benefit > > from the feature in the future. It dose not means WarpDrive is useless before > > that. And it works for our Zip and RSA accelerators in physical world. > > Just not with random process address ... > > > > If you still want non SVA/SVM path what you want to do only works > > > if both ptr1 and ptr2 are in a range that is DMA mapped to the > > > device (moreover you need DMA address to match process address > > > which is not an easy feat). > > > > > > Now even if you only want SVA/SVM, i do not see what is the point > > > of doing this inside VFIO. AMD GPU driver does not and there would > > > be no benefit for them to be there. Well a AMD VFIO mdev device > > > driver for QEMU guest might be useful but they have SVIO IIRC. > > > > > > For SVA/SVM your usage model is: > > > > > > Setup: > > > - user space create a warp drive context for the process > > > - user space create a device specific context for the process > > > - user space create a user space command queue for the device > > > - user space bind command queue > > > > > > At this point the kernel driver has bound the process address > > > space to the device with a command queue and userspace > > > > > > Usage: > > > - user space schedule work and call appropriate flush/update > > > ioctl from time to time. Might be optional depends on the > > > hardware, but probably a good idea to enforce so that kernel > > > can unbind the command queue to bind another process command > > > queue. > > > ... > > > > > > Cleanup: > > > - user space unbind command queue > > > - user space destroy device specific context > > > - user space destroy warp drive context > > > All the above can be implicit when closing the device file. > > > > > > So again in the above model i do not see anywhere something from > > > VFIO that would benefit this model. > > > > > > > Let me show you how the model will be if I use VFIO: > > > > Setup (Kernel part) > > - Kernel driver do every as usual to serve the other functionality, NIC > > can still be registered to netdev, encryptor can still be registered > > to crypto... > > - At the same time, the driver can devote some of its hardware resource > > and register them as a mdev creator to the VFIO framework. This just > > need limited change to the VFIO type1 driver. > > In the above VFIO does not help you one bit ... you can do that with > as much code with new common device as front end. > > > Setup (User space) > > - System administrator create mdev via the mdev creator interface. > > - Following VFIO setup routine, user space open the mdev's group, there is > > only one group for one device. > > - Without PASID support, you don't need to do anything. With PASID, bind > > the PASID to the device via VFIO interface. > > - Get the device from the group via VFIO interface and mmap it the user > > space for device's MMIO access (for the queue). > > - Map whatever memory you need to share with the device with VFIO > > interface. > > - (opt) Add more devices into the container if you want to share the > > same address space with them > > So all VFIO buys you here is boiler plate code that does insert_pfn() > to handle MMIO mapping. Which is just couple hundred lines of boiler > plate code. > No. With VFIO, I don't need to: 1. GUP and accounting for RLIMIT_MEMLOCK 2. Keep all GUP pages for releasing (VFIO uses the rb_tree to do so) 2. Handle the PASID on SMMU (ARM's IOMMU) myself. 3. Multiple devices menagement (VFIO uses container to manage this) And even as a boiler plate, it is valueable, the memory thing is sensitive interface to user space, it can easily become a security problem. If I can achieve my target within the scope of VFIO, why not? At lease it has been proved to be safe for the time being. > > > > Cleanup: > > - User space close the group file handler > > - There will be a problem to let the other process know the mdev is > > freed to be used again. My RFCv1 choose a file handler solution. Alex > > dose not like it. But it is not a big problem. We can always have a > > scheduler process to manage the state of the mdev or even we can > > switch back to the RFCv1 solution without too much effort if we like > > in the future. > > If you were outside VFIO you would have more freedom on how to do that. > For instance process opening the device file can be placed on queue and > first one in the queue get to use the device until it closes/release the > device. Then next one in queue get the device ... Yes. I do like the file handle solution. But I hope the solution become mature as soon as possible. Many of our products, and as I know include some of our partners, are waiting for a long term solution as direction. If I rely on some unmature solution, they may choose some deviated, customized solution. That will be much harmful. Compare to this, the freedom is not so important... > > > Except for the minimum update to the type1 driver and use sdmdev to manage the > > interrupt sharing, I don't need any extra code to gain the address sharing > > capability. And the capability will be strengthen along with the upgrade of VFIO. > > > > > > > > > > > And I don't understand why I should avoid to use VFIO? As Alex said, VFIO is the > > > > > > user driver framework. And I need exactly a user driver interface. Why should I > > > > > > invent another wheel? It has most of stuff I need: > > > > > > > > > > > > 1. Connecting multiple devices to the same application space > > > > > > 2. Pinning and DMA from the application space to the whole set of device > > > > > > 3. Managing hardware resource by device > > > > > > > > > > > > We just need the last step: make sure multiple applications and the kernel can > > > > > > share the same IOMMU. Then why shouldn't we use VFIO? > > > > > > > > > > Because tons of other drivers already do all of the above outside VFIO. Many > > > > > driver have a sizeable userspace side to them (anything with ioctl do) so they > > > > > can be construded as userspace driver too. > > > > > > > > > > > > > Ignoring if there are *tons* of drivers are doing that;), even I do the same as > > > > i915 and solve the address space problem. And if I don't need to with VFIO, why > > > > should I spend so much effort to do it again? > > > > > > Because you do not need any code from VFIO, nor do you need to reinvent > > > things. If non SVA/SVM matters to you then use dma buffer. If not then > > > i do not see anything in VFIO that you need. > > > > > > > As I have explain, if I don't use VFIO, at lease I have to do all that has been > > done in i915 or even more than that. > > So beside the MMIO mmap() handling and dma mapping of range of user space > address space (again all very boiler plate code duplicated accross the > kernel several time in different forms). You do not gain anything being > inside VFIO right ? > As I said, rb-tree for gup, rlimit accounting, cooperation on SMMU, and mature user interface are our concern. > > > > > > So there is no reasons to do that under VFIO. Especialy as in your example > > > > > it is not a real user space device driver, the userspace portion only knows > > > > > about writting command into command buffer AFAICT. > > > > > > > > > > VFIO is for real userspace driver where interrupt, configurations, ... ie > > > > > all the driver is handled in userspace. This means that the userspace have > > > > > to be trusted as it could program the device to do DMA to anywhere (if > > > > > IOMMU is disabled at boot which is still the default configuration in the > > > > > kernel). > > > > > > > > > > > > > But as Alex explained, VFIO is not simply used by VM. So it need not to have all > > > > stuffs as a driver in host system. And I do need to share the user space as DMA > > > > buffer to the hardware. And I can get it with just a little update, then it can > > > > service me perfectly. I don't understand why I should choose a long route. > > > > > > Again this is not the long route i do not see anything in VFIO that > > > benefit you in the SVA/SVM case. A basic character device driver can > > > do that. > > > > > > > > > > > So i do not see any reasons to do anything you want inside VFIO. All you > > > > > want to do can be done outside as easily. Moreover it would be better if > > > > > you define clearly each scenario because from where i sit it looks like > > > > > you are opening the door wide open to userspace to DMA anywhere when IOMMU > > > > > is disabled. > > > > > > > > > > When IOMMU is disabled you can _not_ expose command queue to userspace > > > > > unless your device has its own page table and all commands are relative > > > > > to that page table and the device page table is populated by kernel driver > > > > > in secure way (ie by checking that what is populated can be access). > > > > > > > > > > I do not believe your example device to have such page table nor do i see > > > > > a fallback path when IOMMU is disabled that force user to do ioctl for > > > > > each commands. > > > > > > > > > > Yes i understand that you target SVA/SVM but still you claim to support > > > > > non SVA/SVM. The point is that userspace can not be trusted if you want > > > > > to have random program use your device. I am pretty sure that all user > > > > > of VFIO are trusted process (like QEMU). > > > > > > > > > > > > > > > Finaly i am convince that the IOMMU grouping stuff related to VFIO is > > > > > useless for your usecase. I really do not see the point of that, it > > > > > does complicate things for you for no reasons AFAICT. > > > > > > > > Indeed, I don't like the group thing. I believe VFIO's maintains would not like > > > > it very much either;). But the problem is, the group reflects to the same > > > > IOMMU(unit), which may shared with other devices. It is a security problem. I > > > > cannot ignore it. I have to take it into account event I don't use VFIO. > > > > > > To me it seems you are making a policy decission in kernel space ie > > > wether the device should be isolated in its own group or not is a > > > decission that is up to the sys admin or something in userspace. > > > Right now existing user of SVA/SVM don't (at least AFAICT). > > > > > > Do we really want to force such isolation ? > > > > > > > But it is not my decision, that how the iommu subsystem is designed. Personally > > I don't like it at all, because all our hardwares have their own stream id > > (device id). I don't need the group concept at all. But the iommu subsystem > > assume some devices may share the name device ID to a single IOMMU. > > My question was do you really want to force group isolation for the > device ? Existing SVA/SVM capable driver do not force that, they let > the userspace decide this (sysadm, distributions, ...). Being part of > VFIO (in the way you do, likely ways to avoid this inside VFIO too) > force this decision ie make a policy decision without userspace having > anything to say about it. > > > The IOMMU group thing as always been doubt full to me, it is advertise > as allowing to share resources (ie IOMMU page table) between devices. > But this assume that all device driver in the group have some way of > communicating with each other to share common DMA address that point > to memory devices care. I believe only VFIO does that and probably > only when use by QEMU. > > > Anyway my question is: > > Is it that much useful to be inside VFIO (to avoid few hundred lines > of boiler plate code) given that it forces you into a model (group > isolation) that so far have never been the prefered way for all > existing device driver that already do what you want to achieve ? > You mean to say I create another framework and copy most of the code from VFIO? It is hard to believe the mainline kernel will take my code. So how about let me try the VFIO way first and try that if it won't work? ;) > > >From where i stand i do not see overwhelming reasons to do what you > are doing inside VFIO. > > To me it would make more sense to have regular device driver. They > all can have device file under same hierarchy to make devices with > same programming model easy to discover. > > Jérôme Cheers -- -Kenneth(Hisilicon) ================================================================================ 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁 止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中 的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!