From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dalessandro, Dennis" Subject: Re: [PATCHv12 1/3] rdmacg: Added rdma cgroup controller Date: Mon, 19 Sep 2016 13:10:23 +0000 Message-ID: <1474290618.6520.23.camel@intel.com> References: <20160901084406.GA4115@lst.de> <20160910161442.GC29259@lst.de> <20160910170151.GA5230@obsidianresearch.com> <20160911133421.GA23384@lst.de> <20160911143522.GL6415@leon.nu> <20160911171409.GA13442@obsidianresearch.com> <20160911172445.GA25953@lst.de> <20160911175235.GB13442@obsidianresearch.com> <20160912050717.GE8812@leon.nu> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: Content-Language: en-US Content-ID: <584812C046C96144905E4C617261D9D5-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Sender: cgroups-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: "pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" , "leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org" Cc: "lizefan-hv44wF8Li93QT0dZR+AlfA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "corbet-T1hC0tSOHrs@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org" , "hch-jcswGhMUV9g@public.gmane.org" , "linux-security-module-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "haggaie-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org" , "hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org" , "Hefty, Sean" , "akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org" , "james.l.morris-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org" , "tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org" , "liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org" , "jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org" List-Id: linux-rdma@vger.kernel.org T24gV2VkLCAyMDE2LTA5LTE0IGF0IDEyOjM2ICswNTMwLCBQYXJhdiBQYW5kaXQgd3JvdGU6DQo+ IEhpIERlbm5pcywNCj4gDQo+IERvIHlvdSBrbm93IGhvdyB3b3VsZCBIRkkxIGRyaXZlciB3b3Vs ZCB3b3JrIGFsb25nIHdpdGggcmRtYSBjZ3JvdXA/DQoNCktlZXAgaW4gbWluZCBIRkkxIGRyaXZl ciBoYXMgdHdvICJtb2RlcyIgb2Ygb3BlcmF0aW9uLiBXZSBzdXBwb3J0DQp2ZXJicywgYW5kIHdv dWxkIHN1cmVseSBmYWxsIGluIGxpbmUgd2l0aCB3aGF0ZXZlciBjZ3JvdXBzIGRvIGZvciBJQg0K Y29yZS4gRm9yIG91ciBwc20gaW50ZXJmYWNlLCBub3Qgc3VyZSBob3cgY2dyb3VwcyB3b3VsZCBj b21lIGludG8gcGxheS4NClBzbSBpcyBkZXNpZ25lZCB0byBleHBvc2UgdGhlIGh3IHRvIHVzZXIg YW5kIGF2b2lkIHRoZSBrZXJuZWwgd2hlbg0KcG9zc2libGUgYWRkaW5nIG1vcmUga2VybmVsIGNv bnRyb2wgaXMgc29ydCBvZiBjb250cmFyeSB0byB0aGF0Lg0KDQpOb3cgdGhhdCBiZWluZyBzYWlk LCBDaHJpc3RvcGggcmVjZW50bHkgbWFkZSBtZW50aW9uIG9mIG1heWJlIGhhdmluZyBhDQpkcml2 ZXJzL3BzbSBbMV0uIEkgcmVhbGx5IGhhdmVuJ3QgaGFkIGEgY2hhbmNlIHRvIHRoaW5rIGFib3V0 IHRoZQ0KaW1wbGljYXRpb25zIG9mIHRoYXQsIGJ1dCBtYXliZSBpdCdzIHdvcnRoIGNvbnNpZGVy aW5nLCBhZnRlciBhbGwgd2UNCmhhdmUgdHdvIGltcGxlbWVudGF0aW9ucywgcWliIGFuZCBoZmkx LiBTbyBhbnl3YXkgSSdtIG5vdCBzdXJlIHdlIG5lZWQNCnRvIGJlIHRvbyBjb25jZXJuZWQgYWJv dXQgY2dyb3VwcyByaWdodCBub3cgYXMgZmFyIGFzIHBzbSBzaWRlIG9mDQp0aGluZ3MgZ29lcy4N Cg0KRGVwZW5kaW5nIGhvdyB0aGluZ3Mgc2hha2Ugb3V0IGZvciB0aGUgdUFQSSByZXdyaXRlLCBv ciB2ZXJicyAyLjAgb3INCndoYXRldmVyIHdlIGFyZSBjYWxsaW5nIGl0IHRvZGF5IHRoaW5ncyBt YXkgY2hhbmdlLg0KDQpbMV3CoGh0dHA6Ly9tYXJjLmluZm8vP2w9bGludXgtcmRtYSZtPTE0NzQw MTcxNDMxMzgzMSZ3PTINCg0KLURlbm55DQoNCj4gSGkgTWF0YW4sIExlb24sIEphc29uLA0KPiBB cGFydCBmcm9tIEhGSTEsIGlzIHRoZXJlIGFueSBvdGhlciBjb25jZXJuPw0KPiBPciBQYXRjaCBp cyBnb29kIHRvIGdvPw0KPiANCj4gNC44IGRhdGVzIGFyZSBjbG9zZSBieSAoMiB3ZWVrcykgYW5k IHRoZXJlIGFyZSB0d28gZ2l0IHRyZWVzIGludm9sdmVkDQo+ICh0aGF0IG1pZ2h0IGNhdXNlIG1l cmdlIGVycm9yIHRvIExpbnVzKSBzbyBpZiB0aGVyZSBhcmUgbm8gaXNzdWVzLCBJDQo+IHdvdWxk IGxpa2UgdG8gbWFrZSByZXF1ZXN0IHRvIERvdWcgdG8gY29uc2lkZXIgaXQgZm9yIDQuOCBlYXJs eSBvbi4NCj4gDQo+IFBhcmF2DQo+IA0KPiBPbiBNb24sIFNlcCAxMiwgMjAxNiBhdCAxMDozNyBB TSwgTGVvbiBSb21hbm92c2t5IDxsZW9uQGtlcm5lbC5vcmc+DQo+IHdyb3RlOg0KPiA+IE9uIFN1 biwgU2VwIDExLCAyMDE2IGF0IDExOjUyOjM1QU0gLTA2MDAsIEphc29uIEd1bnRob3JwZSB3cm90 ZToNCj4gPiA+IE9uIFN1biwgU2VwIDExLCAyMDE2IGF0IDA3OjI0OjQ1UE0gKzAyMDAsIENocmlz dG9waCBIZWxsd2lnDQo+ID4gPiB3cm90ZToNCj4gPiA+ID4gPiA+ID4gSSd2ZSBwb3N0ZWQgc29t ZSBpbml0aWFsIHdvcmsgdG93YXJkIGEpIGEgd2hpbGUgYWdvLCBhbmQNCj4gPiA+ID4gPiA+ID4g b25jZSB3ZQ0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IERpZCBpdCBnZXQgbWVyZ2VkPyBEbyB5b3Ug aGF2ZSBhIHBvaW50ZXI/DQo+ID4gPiA+IA0KPiA+ID4gPiBodHRwOi8vd3d3LnNwaW5pY3MubmV0 L2xpc3RzL2xpbnV4LXJkbWEvbXNnMzE5NTguaHRtbA0KPiA+ID4gDQo+ID4gPiBSaWdodCwgSSBy ZW1lbWJlciB0aGF0LiBDZXJ0YWlubHkgdGhlIHJpZ2h0IGRpcmVjdGlvbg0KPiA+ID4gDQo+ID4g PiA+ID4gSG93ZXZlciwgZXZlcnl0aGluZyB1bmRlciB2ZXJicyBpcyBub3Qgc3RyYWlnaHRmb3J3 YXJkLiBUaGUNCj4gPiA+ID4gPiBmaWxlcyBpbg0KPiA+ID4gPiA+IHVzZXJzcGFjZSBhcmUgbm90 IGNvcGllcy4uLg0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IHVzZXI6DQo+ID4gPiA+ID4gDQo+ID4g PiA+ID4gc3RydWN0IGlidl9xdWVyeV9kZXZpY2Ugew0KPiA+ID4gPiA+IMKgwqDCoMKgwqDCoMKg X191MzIgY29tbWFuZDsNCj4gPiA+ID4gPiDCoMKgwqDCoMKgwqDCoF9fdTE2IGluX3dvcmRzOw0K PiA+ID4gPiA+IMKgwqDCoMKgwqDCoMKgX191MTYgb3V0X3dvcmRzOw0KPiA+ID4gPiA+IMKgwqDC oMKgwqDCoMKgX191NjQgcmVzcG9uc2U7DQo+ID4gPiA+ID4gwqDCoMKgwqDCoMKgwqBfX3U2NCBk cml2ZXJfZGF0YVswXTsNCj4gPiA+ID4gPiB9Ow0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IGtlcm5l bDoNCj4gPiA+ID4gPiANCj4gPiA+ID4gPiBzdHJ1Y3QgaWJfdXZlcmJzX3F1ZXJ5X2RldmljZSB7 DQo+ID4gPiA+ID4gwqDCoMKgwqDCoMKgwqDCoF9fdTY0IHJlc3BvbnNlOw0KPiA+ID4gPiA+IMKg wqDCoMKgwqDCoMKgwqBfX3U2NCBkcml2ZXJfZGF0YVswXTsNCj4gPiA+ID4gPiB9Ow0KPiA+ID4g PiANCj4gPiA+ID4gV2UnbGwgb2J2aW91c2x5IG5lZWQgZGlmZmVyZW50IHN0cnV0dXJlcyBmb3Ig dGhlIGxpYmlidmVycyBBUEkNCj4gPiA+ID4gYW5kIHRoZSBrZXJuZWwgaW50ZXJmYWNlIGluIHRo aXMgY2FzZSwgYW5kIHdlJ2xsIG5lZWQgdG8gZmlndXJlDQo+ID4gPiA+IG91dA0KPiA+ID4gPiBo b3cgdG8gcHJvcGVybHkgdHJhbnNsYXRlIHRoZW0uwqDCoEkgdGhpbmsgYSBjYXN0LCBwbHVzIGNv bXBpbGUNCj4gPiA+ID4gdGltZQ0KPiA+ID4gPiB0eXBlIGNoZWNraW5nIGFsYSBCVUlMRF9CVUdf T04gaXMgdGhlIHdheSB0byBnby4NCj4gPiA+IA0KPiA+ID4gSSdtIG5vdCBzdXJlIEkgZm9sbG93 LCB3aGljaCB3b3VsZCBJIGNhc3Q/DQo+ID4gPiANCj4gPiA+IEJVSUxEX0JVR19PTihzaXplb2Yo aWJ2X3F1ZXJ5X2RldmljZSkgPT0NCj4gPiA+IHNpemVvZihpYl91dmVyYnNfY21kX2hkcikgKw0K PiA+ID4gwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBzaXplb2YoaWJfdXZlcmJzX3F1ZXJ5X2Rl dmljZSkpDQo+ID4gPiANCj4gPiA+ID8NCj4gPiA+IA0KPiA+ID4gPiA+IEknbSB0aGlua2luZyB0 aGUgYmVzdCB3YXkgZm9yd2FyZCBtaWdodCBiZSB0byB1c2UgYSBzY3JpcHQNCj4gPiA+ID4gPiBh bmQNCj4gPiA+ID4gPiB0cmFuc2Zvcm0gdXNlcnNwYWNlIGludG86DQo+ID4gPiA+ID4gDQo+ID4g PiA+ID4gc3RydWN0IGlidl9xdWVyeV9kZXZpY2Ugew0KPiA+ID4gPiA+IMKgIHN0cnVjdCBpYl91 dmVyYnNfY21kX2hkciBoZHI7DQo+ID4gPiA+ID4gwqAgc3RydWN0IGliX3V2ZXJic19xdWVyeV9k ZXZpY2UgY21kOw0KPiA+ID4gPiA+IH07DQo+ID4gPiA+IA0KPiA+ID4gPiBUaGF0IHdvdWxkIGJy ZWFrIHRoZSB1c2VycyBvZiB0aGUgaW50ZXJmYWNlLg0KPiA+ID4gDQo+ID4gPiBTb3JyeSwgSSBt ZWFuIGRvaW5nIHRoaXMgaW5zaWRlIHJkbWEtcGx1bWJpbmcuIFNpbmNlIHRoZSBjaGFuZ2UNCj4g PiA+IGlzIEFCSQ0KPiA+ID4gaWRlbnRpY2FsIHRoZSBtb2RpZmllZCBsaWJpYnZlcmJzIHdvdWxk IHN0aWxsIGJlIGJpbmFyeQ0KPiA+ID4gY29tcGF0aWJsZQ0KPiA+ID4gd2l0aCBhbGwgcHJvdmlk ZXJzIGJ1dCBub3Qgc291cmNlIGNvbXBhdGlibGUuIFNpbmNlIGFsbCBrZXJuZWwNCj4gPiA+IHN1 cHBvcnRlZCBwcm92aWRlcnMgYXJlIGluIHJkbWEtcGx1bWJpbmcgd2UgY2FuIGFkZCB0aGUgJy5j bWQuJw0KPiA+ID4gYXQgdGhlDQo+ID4gPiBzYW1lIHRpbWUuDQo+ID4gPiANCj4gPiA+IFRoZSBr ZXJuZWwgdWFwaSBoZWFkZXIgd291bGQgc3RheSB0aGUgc2FtZS4NCj4gPiA+IA0KPiA+ID4gPiBI b3dldmVyIGF1dG9tYXRpY2FsbHkgZ2VuZXJhdGluZyB0aGUgdXNlciBBQkkgZnJvbSB0aGUga2Vy bmVsDQo+ID4gPiA+IG9uZQ0KPiA+ID4gPiBtaWdodCBzdGlsbCBiZSBhIGdvb2QgaWRlYSBpbiB0 aGUgbG9uZyBydW4uDQo+ID4gPiANCj4gPiA+IE15IHByZWZlcmVuY2Ugd291bGQgYmUgdG8gdHJ5 IGFuZCB1c2UgdGhlIGtlcm5lbCBoZWFkZXJzDQo+ID4gPiBkaXJlY3RseS4NCj4gPiANCj4gPiBJ IHRob3VnaHQgdGhlIHNhbWUsIGVzcGVjaWFsbHkgYWZ0ZXIgcmVhbGl6aW5nIHRoYXQgdGhleSBh cmUgYWxtb3N0DQo+ID4gY29weS9wYXN0ZSBmcm9tIHRoZSB2ZW5kb3IgKi1hYmkuaCBmaWxlcy4N Cj4gPiANCj4gPiA+IA0KPiA+ID4gSmFzb24NCj4gLS0NCj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0 aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGxpbnV4LXJkbWEiIA0KPiBpbg0K PiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZnZXIua2VybmVsLm9yZw0KPiBN b3JlIG1ham9yZG9tbyBpbmZvIGF0wqDCoGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21v LWluZm8uaHRtbA== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S938987AbcISNYo (ORCPT ); Mon, 19 Sep 2016 09:24:44 -0400 Received: from mga06.intel.com ([134.134.136.31]:46887 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S938584AbcISNK3 (ORCPT ); Mon, 19 Sep 2016 09:10:29 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.30,361,1470726000"; d="scan'208";a="11293123" From: "Dalessandro, Dennis" To: "pandit.parav@gmail.com" , "leon@kernel.org" CC: "lizefan@huawei.com" , "linux-kernel@vger.kernel.org" , "corbet@lwn.net" , "linux-rdma@vger.kernel.org" , "cgroups@vger.kernel.org" , "ogerlitz@mellanox.com" , "hch@lst.de" , "linux-security-module@vger.kernel.org" , "haggaie@mellanox.com" , "hannes@cmpxchg.org" , "Hefty, Sean" , "akpm@linux-foundation.org" , "james.l.morris@oracle.com" , "tj@kernel.org" , "liranl@mellanox.com" , "jgunthorpe@obsidianresearch.com" , "linux-doc@vger.kernel.org" , "dledford@redhat.com" , "matanb@mellanox.com" , "serge@hallyn.com" Subject: Re: [PATCHv12 1/3] rdmacg: Added rdma cgroup controller Thread-Topic: [PATCHv12 1/3] rdmacg: Added rdma cgroup controller Thread-Index: AQHSA2N02GqpcdQRok22e1lYG1qYXqBjk2wbgADAlq+ACcQfgIAAD8gAgAUyxQCAAA0sgIABWFyAgAARDACAACxdgIAAAvaAgAAHx4CAALyCgIADReyAgAhBWgA= Date: Mon, 19 Sep 2016 13:10:23 +0000 Message-ID: <1474290618.6520.23.camel@intel.com> References: <20160901084406.GA4115@lst.de> <20160910161442.GC29259@lst.de> <20160910170151.GA5230@obsidianresearch.com> <20160911133421.GA23384@lst.de> <20160911143522.GL6415@leon.nu> <20160911171409.GA13442@obsidianresearch.com> <20160911172445.GA25953@lst.de> <20160911175235.GB13442@obsidianresearch.com> <20160912050717.GE8812@leon.nu> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.252.176.147] Content-Type: text/plain; charset="utf-8" Content-ID: <584812C046C96144905E4C617261D9D5@intel.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id u8JDOs0H025589 On Wed, 2016-09-14 at 12:36 +0530, Parav Pandit wrote: > Hi Dennis, > > Do you know how would HFI1 driver would work along with rdma cgroup? Keep in mind HFI1 driver has two "modes" of operation. We support verbs, and would surely fall in line with whatever cgroups do for IB core. For our psm interface, not sure how cgroups would come into play. Psm is designed to expose the hw to user and avoid the kernel when possible adding more kernel control is sort of contrary to that. Now that being said, Christoph recently made mention of maybe having a drivers/psm [1]. I really haven't had a chance to think about the implications of that, but maybe it's worth considering, after all we have two implementations, qib and hfi1. So anyway I'm not sure we need to be too concerned about cgroups right now as far as psm side of things goes. Depending how things shake out for the uAPI rewrite, or verbs 2.0 or whatever we are calling it today things may change. [1] http://marc.info/?l=linux-rdma&m=147401714313831&w=2 -Denny > Hi Matan, Leon, Jason, > Apart from HFI1, is there any other concern? > Or Patch is good to go? > > 4.8 dates are close by (2 weeks) and there are two git trees involved > (that might cause merge error to Linus) so if there are no issues, I > would like to make request to Doug to consider it for 4.8 early on. > > Parav > > On Mon, Sep 12, 2016 at 10:37 AM, Leon Romanovsky > wrote: > > On Sun, Sep 11, 2016 at 11:52:35AM -0600, Jason Gunthorpe wrote: > > > On Sun, Sep 11, 2016 at 07:24:45PM +0200, Christoph Hellwig > > > wrote: > > > > > > > I've posted some initial work toward a) a while ago, and > > > > > > > once we > > > > > > > > > > Did it get merged? Do you have a pointer? > > > > > > > > http://www.spinics.net/lists/linux-rdma/msg31958.html > > > > > > Right, I remember that. Certainly the right direction > > > > > > > > However, everything under verbs is not straightforward. The > > > > > files in > > > > > userspace are not copies... > > > > > > > > > > user: > > > > > > > > > > struct ibv_query_device { > > > > >        __u32 command; > > > > >        __u16 in_words; > > > > >        __u16 out_words; > > > > >        __u64 response; > > > > >        __u64 driver_data[0]; > > > > > }; > > > > > > > > > > kernel: > > > > > > > > > > struct ib_uverbs_query_device { > > > > >         __u64 response; > > > > >         __u64 driver_data[0]; > > > > > }; > > > > > > > > We'll obviously need different strutures for the libibvers API > > > > and the kernel interface in this case, and we'll need to figure > > > > out > > > > how to properly translate them.  I think a cast, plus compile > > > > time > > > > type checking ala BUILD_BUG_ON is the way to go. > > > > > > I'm not sure I follow, which would I cast? > > > > > > BUILD_BUG_ON(sizeof(ibv_query_device) == > > > sizeof(ib_uverbs_cmd_hdr) + > > >              sizeof(ib_uverbs_query_device)) > > > > > > ? > > > > > > > > I'm thinking the best way forward might be to use a script > > > > > and > > > > > transform userspace into: > > > > > > > > > > struct ibv_query_device { > > > > >   struct ib_uverbs_cmd_hdr hdr; > > > > >   struct ib_uverbs_query_device cmd; > > > > > }; > > > > > > > > That would break the users of the interface. > > > > > > Sorry, I mean doing this inside rdma-plumbing. Since the change > > > is ABI > > > identical the modified libibverbs would still be binary > > > compatible > > > with all providers but not source compatible. Since all kernel > > > supported providers are in rdma-plumbing we can add the '.cmd.' > > > at the > > > same time. > > > > > > The kernel uapi header would stay the same. > > > > > > > However automatically generating the user ABI from the kernel > > > > one > > > > might still be a good idea in the long run. > > > > > > My preference would be to try and use the kernel headers > > > directly. > > > > I thought the same, especially after realizing that they are almost > > copy/paste from the vendor *-abi.h files. > > > > > > > > Jason > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" > in > the body of a message to majordomo@vger.kernel.org > More majordomo info at  http://vger.kernel.org/majordomo-info.html