From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kenny Ho Subject: Re: [RFC PATCH 0/5] cgroup support for GPU devices Date: Sun, 5 May 2019 10:21:30 -0400 Message-ID: References: <20190501140438.9506-1-brian.welty@intel.com> <20190502083433.GP7676@mtr-leonro.mtl.com> <20190505071436.GD6938@mtr-leonro.mtl.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20190505071436.GD6938@mtr-leonro.mtl.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Leon Romanovsky Cc: Parav Pandit , David Airlie , kenny.ho@amd.com, intel-gfx@lists.freedesktop.org, "Welty, Brian" , cgroups@vger.kernel.org, dri-devel@lists.freedesktop.org, Michal Hocko , Johannes Weiner , linux-mm@kvack.org, J??r??me Glisse , Li Zefan , Vladimir Davydov , Alex Deucher , Tejun Heo , Christian K??nig , RDMA mailing list List-Id: linux-rdma@vger.kernel.org T24gU3VuLCBNYXkgNSwgMjAxOSBhdCAzOjE0IEFNIExlb24gUm9tYW5vdnNreSA8bGVvbkBrZXJu ZWwub3JnPiB3cm90ZToKPiA+ID4gRG9lc24ndCBSRE1BIGFscmVhZHkgaGFzIGEgc2VwYXJhdGUg Y2dyb3VwPyAgV2h5IG5vdCBpbXBsZW1lbnQgaXQgdGhlcmU/Cj4gPiA+Cj4gPgo+ID4gSGkgS2Vu bnksIEkgY2FuJ3QgYW5zd2VyIGZvciBMZW9uLCBidXQgSSdtIGhvcGVmdWwgaGUgYWdyZWVzIHdp dGggcmF0aW9uYWxlCj4gPiBJIGdhdmUgaW4gdGhlIGNvdmVyIGxldHRlci4gIE5hbWVseSwgdG8g aW1wbGVtZW50IGluIHJkbWEgY29udHJvbGxlciwgd291bGQKPiA+IG1lYW4gZHVwbGljYXRpbmcg ZXhpc3RpbmcgbWVtY2cgY29udHJvbHMgdGhlcmUuCj4KPiBFeGFjdGx5LCBJIGRpZG4ndCBmZWVs IGNvbWZvcnRhYmxlIHRvIGFkZCBub3Rpb24gb2YgImRldmljZSBtZW1vcnkiCj4gdG8gUkRNQSBj Z3JvdXAgYW5kIHBvc3Rwb25lZCB0aGF0IGRlY2lzaW9uIHRvIGxhdGVyIHBvaW50IG9mIHRpbWUu Cj4gUkRNQSBvcGVyYXRlcyB3aXRoIHZlcmJzIG9iamVjdHMgYW5kIGFsbCBvdXIgdXNlciBzcGFj ZSBBUEkgaXMgYmFzZWQgYXJvdW5kCj4gdGhhdCBjb25jZXB0LiBBdCB0aGUgZW5kLCBzeXN0ZW0g YWRtaW5pc3RyYXRvciB3aWxsIGhhdmUgaGFyZCB0aW1lIHRvCj4gdW5kZXJzdGFuZCB0aGUgZGlm ZmVyZW5jZXMgYmV0d2VlbiBtZW1jZyBhbmQgUkRNQSBtZW1vcnkuCkludGVyZXN0aW5nLiAgSSBh Y3R1YWxseSBkb24ndCB1bmRlcnN0YW5kIHRoaXMgcGFydCAoSSB3b3JrZWQgaW4KZGV2b3BzL3N5 c2FkbWluIHNpZGUgb2YgdGhpbmdzIGJ1dCBuZXZlciB3aXRoIHJkbWEuKSAgRG9uJ3QKYXBwbGlj YXRpb25zIHRoYXQgdXNlIHJkbWEgcmVxdWlyZSBzb21lIGF3YXJlbmVzcyBvZiByZG1hIChJIG1l YW4sIHlvdQptZW50aW9uZWQgdmVyYnMgYW5kIG9iamVjdHMuLi4gb3IgZG8gdGhleSBqdXN0IHVz ZSByZWd1bGFyIG1hbGxvYyBmb3IKYnVmZmVyIGFsbG9jYXRpb24gYW5kIHRoZW4gc2VuZCBpdCB0 aHJvdWdoIHNvbWUgZnVuY3Rpb24/KSAgQXMgYSB1c2VyLApJIHdvdWxkIGhhdmUgdGhpcyBxdWVz dGlvbjogd2h5IGRvIEkgbmVlZCB0byBjb25maWd1cmUgc29tZSBwYXJ0IG9mCnJkbWEgcmVzb3Vy Y2VzIHVuZGVyIHJkbWEgY2dyb3VwIHdoaWxlIG90aGVyIHBhcnQgb2YgcmRtYSByZXNvdXJjZXMg aW4KYSBkaWZmZXJlbnQsIHNlZW1pbmdseSB1bnJlbGF0ZWQgY2dyb3Vwcy4KCkkgdGhpbmsgd2Ug bmVlZCB0byBiZSBjYXJlZnVsIGFib3V0IGRyYXdpbmcgdGhlIGxpbmUgYmV0d2VlbgpkdXBsaWNh dGlvbiBhbmQgb3ZlciBjb3VwbGluZ3MgYmV0d2VlbiBzdWJzeXN0ZW1zLiAgSSBoYXZlIG90aGVy CnRob3VnaHRzIGFuZCBjb25jZXJucyBhbmQgSSB3aWxsIHRyeSB0byBvcmdhbml6ZSB0aGVtIGlu dG8gYSByZXNwb25zZQppbiB0aGUgbmV4dCBmZXcgZGF5cy4KClJlZ2FyZHMsCktlbm55CgoKPiA+ Cj4gPiBJcyBBTUQgaW50ZXJlc3RlZCBpbiBjb2xsYWJvcmF0aW5nIHRvIGhlbHAgc2hhcGUgdGhp cyBmcmFtZXdvcms/Cj4gPiBJdCBpcyBpbnRlbmRlZCB0byBiZSBkZXZpY2UtbmV1dHJhbCwgc28g Y291bGQgYmUgbGV2ZXJhZ2VkIGJ5IHZhcmlvdXMKPiA+IHR5cGVzIG9mIGRldmljZXMuCj4gPiBJ ZiB5b3UgaGF2ZSBhbiBhbHRlcm5hdGl2ZSBzb2x1dGlvbiB3ZWxsIHVuZGVyd2F5LCB0aGVuIG1h eWJlCj4gPiB3ZSBjYW4gd29yayB0b2dldGhlciB0byBtZXJnZSBvdXIgZWZmb3J0cyBpbnRvIG9u ZS4KPiA+IEluIHRoZSBlbmQsIHRoZSBEUk0gY29tbXVuaXR5IGlzIGJlc3Qgc2VydmVkIHdpdGgg Y29tbW9uIHNvbHV0aW9uLgo+ID4KPiA+Cj4gPiA+Cj4gPiA+Pj4gYW5kIHdpdGggZnV0dXJlIHdv cmssIHdlIGNvdWxkIGV4dGVuZCB0bzoKPiA+ID4+PiAqICB0cmFjayBhbmQgY29udHJvbCBzaGFy ZSBvZiBHUFUgdGltZSAocmV1c2Ugb2YgY3B1L2NwdWFjY3QpCj4gPiA+Pj4gKiAgYXBwbHkgbWFz ayBvZiBhbGxvd2VkIGV4ZWN1dGlvbiBlbmdpbmVzIChyZXVzZSBvZiBjcHVzZXRzKQo+ID4gPj4+ Cj4gPiA+Pj4gSW5zdGVhZCBvZiBpbnRyb2R1Y2luZyBhIG5ldyBjZ3JvdXAgc3Vic3lzdGVtIGZv ciBHUFUgZGV2aWNlcywgYSBuZXcKPiA+ID4+PiBmcmFtZXdvcmsgaXMgcHJvcG9zZWQgdG8gYWxs b3cgZGV2aWNlcyB0byByZWdpc3RlciB3aXRoIGV4aXN0aW5nIGNncm91cAo+ID4gPj4+IGNvbnRy b2xsZXJzLCB3aGljaCBjcmVhdGVzIHBlci1kZXZpY2UgY2dyb3VwX3N1YnN5c19zdGF0ZSB3aXRo aW4gdGhlCj4gPiA+Pj4gY2dyb3VwLiAgVGhpcyBnaXZlcyBkZXZpY2UgZHJpdmVycyB0aGVpciBv d24gcHJpdmF0ZSBjZ3JvdXAgY29udHJvbHMKPiA+ID4+PiAoc3VjaCBhcyBtZW1vcnkgbGltaXRz IG9yIG90aGVyIHBhcmFtZXRlcnMpIHRvIGJlIGFwcGxpZWQgdG8gZGV2aWNlCj4gPiA+Pj4gcmVz b3VyY2VzIGluc3RlYWQgb2YgaG9zdCBzeXN0ZW0gcmVzb3VyY2VzLgo+ID4gPj4+IERldmljZSBk cml2ZXJzIChHUFUgb3Igb3RoZXIpIGFyZSB0aGVuIGFibGUgdG8gcmV1c2UgdGhlIGV4aXN0aW5n IGNncm91cAo+ID4gPj4+IGNvbnRyb2xzLCBpbnN0ZWFkIG9mIGludmVudGluZyBzaW1pbGFyIG9u ZXMuCj4gPiA+Pj4KPiA+ID4+PiBQZXItZGV2aWNlIGNvbnRyb2xzIHdvdWxkIGJlIGV4cG9zZWQg aW4gY2dyb3VwIGZpbGVzeXN0ZW0gYXM6Cj4gPiA+Pj4gICAgIG1vdW50LzxjZ3JvdXBfbmFtZT4v PHN1YnN5c19uYW1lPi5kZXZpY2VzLzxkZXZfbmFtZT4vPHN1YnN5c19maWxlcz4KPiA+ID4+PiBz dWNoIGFzIChmb3IgZXhhbXBsZSk6Cj4gPiA+Pj4gICAgIG1vdW50LzxjZ3JvdXBfbmFtZT4vbWVt b3J5LmRldmljZXMvPGRldl9uYW1lPi9tZW1vcnkubWF4Cj4gPiA+Pj4gICAgIG1vdW50LzxjZ3Jv dXBfbmFtZT4vbWVtb3J5LmRldmljZXMvPGRldl9uYW1lPi9tZW1vcnkuY3VycmVudAo+ID4gPj4+ ICAgICBtb3VudC88Y2dyb3VwX25hbWU+L2NwdS5kZXZpY2VzLzxkZXZfbmFtZT4vY3B1LnN0YXQK PiA+ID4+PiAgICAgbW91bnQvPGNncm91cF9uYW1lPi9jcHUuZGV2aWNlcy88ZGV2X25hbWU+L2Nw dS53ZWlnaHQKPiA+ID4+Pgo+ID4gPj4+IFRoZSBkcm0vaTkxNSBwYXRjaCBpbiB0aGlzIHNlcmll cyBpcyBiYXNlZCBvbiB0b3Agb2Ygb3RoZXIgUkZDIHdvcmsgWzFdCj4gPiA+Pj4gZm9yIGk5MTUg ZGV2aWNlIG1lbW9yeSBzdXBwb3J0Lgo+ID4gPj4+Cj4gPiA+Pj4gQU1EIFsyXSBhbmQgSW50ZWwg WzNdIGhhdmUgcHJvcG9zZWQgcmVsYXRlZCB3b3JrIGluIHRoaXMgYXJlYSB3aXRoaW4gdGhlCj4g PiA+Pj4gbGFzdCBmZXcgeWVhcnMsIGxpc3RlZCBiZWxvdyBhcyByZWZlcmVuY2UuICBUaGlzIG5l dyBSRkMgcmV1c2VzIGV4aXN0aW5nCj4gPiA+Pj4gY2dyb3VwIGNvbnRyb2xsZXJzIGFuZCB0YWtl cyBhIGRpZmZlcmVudCBhcHByb2FjaCB0aGFuIHByaW9yIHdvcmsuCj4gPiA+Pj4KPiA+ID4+PiBG aW5hbGx5LCBzb21lIHBvdGVudGlhbCBkaXNjdXNzaW9uIHBvaW50cyBmb3IgdGhpcyBzZXJpZXM6 Cj4gPiA+Pj4gKiBtZXJnZSBwcm9wb3NlZCA8c3Vic3lzX25hbWU+LmRldmljZXMgaW50byBhIHNp bmdsZSBkZXZpY2VzIGRpcmVjdG9yeT8KPiA+ID4+PiAqIGFsbG93IGRldmljZXMgdG8gaGF2ZSBt dWx0aXBsZSByZWdpc3RyYXRpb25zIGZvciBzdWJzZXRzIG9mIHJlc291cmNlcz8KPiA+ID4+PiAq IGRvY3VtZW50IGEgJ2NvbW1vbiBjaGFyZ2luZyBwb2xpY3knIGZvciBkZXZpY2UgZHJpdmVycyB0 byBmb2xsb3c/Cj4gPiA+Pj4KPiA+ID4+PiBbMV0gaHR0cHM6Ly9wYXRjaHdvcmsuZnJlZWRlc2t0 b3Aub3JnL3Nlcmllcy81NjY4My8KPiA+ID4+PiBbMl0gaHR0cHM6Ly9saXN0cy5mcmVlZGVza3Rv cC5vcmcvYXJjaGl2ZXMvZHJpLWRldmVsLzIwMTgtTm92ZW1iZXIvMTk3MTA2Lmh0bWwKPiA+ID4+ PiBbM10gaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvYXJjaGl2ZXMvaW50ZWwtZ2Z4LzIw MTgtSmFudWFyeS8xNTMxNTYuaHRtbAo+ID4gPj4+Cj4gPiA+Pj4KPiA+ID4+PiBCcmlhbiBXZWx0 eSAoNSk6Cj4gPiA+Pj4gICBjZ3JvdXA6IEFkZCBjZ3JvdXBfc3Vic3lzIHBlci1kZXZpY2UgcmVn aXN0cmF0aW9uIGZyYW1ld29yawo+ID4gPj4+ICAgY2dyb3VwOiBDaGFuZ2Uga2VybmZzX25vZGUg Zm9yIGRpcmVjdG9yaWVzIHRvIHN0b3JlCj4gPiA+Pj4gICAgIGNncm91cF9zdWJzeXNfc3RhdGUK PiA+ID4+PiAgIG1lbWNnOiBBZGQgcGVyLWRldmljZSBzdXBwb3J0IHRvIG1lbW9yeSBjZ3JvdXAg c3Vic3lzdGVtCj4gPiA+Pj4gICBkcm06IEFkZCBtZW1vcnkgY2dyb3VwIHJlZ2lzdHJhdGlvbiBh bmQgRFJJVkVSX0NHUk9VUFMgZmVhdHVyZSBiaXQKPiA+ID4+PiAgIGRybS9pOTE1OiBVc2UgbWVt b3J5IGNncm91cCBmb3IgZW5mb3JjaW5nIGRldmljZSBtZW1vcnkgbGltaXQKPiA+ID4+Pgo+ID4g Pj4+ICBkcml2ZXJzL2dwdS9kcm0vZHJtX2Rydi5jICAgICAgICAgICAgICAgICAgfCAgMTIgKwo+ ID4gPj4+ICBkcml2ZXJzL2dwdS9kcm0vZHJtX2dlbS5jICAgICAgICAgICAgICAgICAgfCAgIDcg Kwo+ID4gPj4+ICBkcml2ZXJzL2dwdS9kcm0vaTkxNS9pOTE1X2Rydi5jICAgICAgICAgICAgfCAg IDIgKy0KPiA+ID4+PiAgZHJpdmVycy9ncHUvZHJtL2k5MTUvaW50ZWxfbWVtb3J5X3JlZ2lvbi5j IHwgIDI0ICstCj4gPiA+Pj4gIGluY2x1ZGUvZHJtL2RybV9kZXZpY2UuaCAgICAgICAgICAgICAg ICAgICB8ICAgMyArCj4gPiA+Pj4gIGluY2x1ZGUvZHJtL2RybV9kcnYuaCAgICAgICAgICAgICAg ICAgICAgICB8ICAgOCArCj4gPiA+Pj4gIGluY2x1ZGUvZHJtL2RybV9nZW0uaCAgICAgICAgICAg ICAgICAgICAgICB8ICAxMSArCj4gPiA+Pj4gIGluY2x1ZGUvbGludXgvY2dyb3VwLWRlZnMuaCAg ICAgICAgICAgICAgICB8ICAyOCArKwo+ID4gPj4+ICBpbmNsdWRlL2xpbnV4L2Nncm91cC5oICAg ICAgICAgICAgICAgICAgICAgfCAgIDMgKwo+ID4gPj4+ICBpbmNsdWRlL2xpbnV4L21lbWNvbnRy b2wuaCAgICAgICAgICAgICAgICAgfCAgMTAgKwo+ID4gPj4+ICBrZXJuZWwvY2dyb3VwL2Nncm91 cC12MS5jICAgICAgICAgICAgICAgICAgfCAgMTAgKy0KPiA+ID4+PiAga2VybmVsL2Nncm91cC9j Z3JvdXAuYyAgICAgICAgICAgICAgICAgICAgIHwgMzEwICsrKysrKysrKysrKysrKysrKy0tLQo+ ID4gPj4+ICBtbS9tZW1jb250cm9sLmMgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAxODMg KysrKysrKysrKystCj4gPiA+Pj4gIDEzIGZpbGVzIGNoYW5nZWQsIDU1MiBpbnNlcnRpb25zKCsp LCA1OSBkZWxldGlvbnMoLSkKPiA+ID4+Pgo+ID4gPj4+IC0tCj4gPiA+Pj4gMi4yMS4wCj4gPiA+ Pj4KPiA+ID4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Cj4gPiA+PiBkcmktZGV2ZWwgbWFpbGluZyBsaXN0Cj4gPiA+PiBkcmktZGV2ZWxAbGlzdHMuZnJl ZWRlc2t0b3Aub3JnCj4gPiA+PiBodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFu L2xpc3RpbmZvL2RyaS1kZXZlbApfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXwpJbnRlbC1nZnggbWFpbGluZyBsaXN0CkludGVsLWdmeEBsaXN0cy5mcmVlZGVz a3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9p bnRlbC1nZng= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5E4EC004C9 for ; Sun, 5 May 2019 14:21:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E994206DF for ; Sun, 5 May 2019 14:21:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AWQlbU8V" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E994206DF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DDC426B0005; Sun, 5 May 2019 10:21:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D65E46B0006; Sun, 5 May 2019 10:21:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDE296B0007; Sun, 5 May 2019 10:21:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by kanga.kvack.org (Postfix) with ESMTP id 87E536B0005 for ; Sun, 5 May 2019 10:21:44 -0400 (EDT) Received: by mail-oi1-f200.google.com with SMTP id m8so3768735oih.0 for ; Sun, 05 May 2019 07:21:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:mime-version:references :in-reply-to:from:date:message-id:subject:to:cc; bh=IQqSIA7uD5M0wQf7Hipk4HlzkvDnQtg0hg1zEHr33zY=; b=pxEOoKQvPS6jKxB5cIEhfg1SyYcu5BXW1UXVz7nSvOky5xmNUuR4a9yStDtDoPMEMD MWObjOIx9T1VpPWNJCLVjjlHWDvS/0MjsUdnf72VAzWblG20mhF+Eg2QDnyl+gj9Jpt5 Gw4grbWZpeygoCnr1el3JGJYMrup8WWjp5gLt0eOdfe0ftdj7x3CoWJs6AxaMhVBrLtC YJDUS5HCEIdZKAGcebyMCOuPSbJYF8eVVuCASvVPH8LJ8OhfKL8A0W1UqVj7vqHDRdjQ d5OsTo9OXkbQqidgrl+vqf/EvhIf+o87Ztn0igoTj+LNd4LM8hYBZza/HzNHWuopsHsC OoEw== X-Gm-Message-State: APjAAAVd8aaw/B+GCYdTqllI2t56UB2JGNMSfd16tKKewngzu+3ybTZF 131pl6SQozbQB0Q3J8ikziAS2Q1ylPkprAFq9gZdT/vOCdPB/4scVbmOyfZwE1h3CrZm4t5Co3y bC0eELpZSBY7nYxQ5j5XBNV7yyuu0tNW9WNhRNCMvT28ika0qXJsTccfa0OekCASltw== X-Received: by 2002:aca:d7d5:: with SMTP id o204mr5515742oig.23.1557066103803; Sun, 05 May 2019 07:21:43 -0700 (PDT) X-Received: by 2002:aca:d7d5:: with SMTP id o204mr5515705oig.23.1557066102855; Sun, 05 May 2019 07:21:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557066102; cv=none; d=google.com; s=arc-20160816; b=JKFcer8e/15GKv3yuqmTjXUOkdgo3CGYTJCQc7OeyZEAMCsbaUj8J70ehHWgQSFoQ5 jiK3g7a5ljfdCUm3tIlRbRH/HGf7oRAgUa07F0t2wKHXTlf458dpI2TXf7bzKPytL1Hu bAWlyp2+Zj2+3LOEcbQVNfcACnFOk2+JbFAPx/EbN60GI4ny/rcyGUZpdcXXECZaEwi+ pMas+EKeLcSs+T7m/7+pyHHMSQyivINqbjgpjO724+e+KVKTaeogF7v1aaiGrxuoe0yG zs5qFS49GeGjVm153kccPFu0FxiAJ7hLtjvMrCTlonKYJOgr6OutjbUaW4ny5RLfl6R0 7wpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=IQqSIA7uD5M0wQf7Hipk4HlzkvDnQtg0hg1zEHr33zY=; b=B/h0XQgnTVAzynbCiNCRT9+5B3OLCBq7wb4ZuFaAPDVQU+26/R867UWbFhq6g5SU+w /RLDXczZIdgipbd3v8WsV50ZiWlEfruZSJDamADqKX8YkdsdC8AEsBg2qyX9LooU33f1 fi1WSPhUepbz8mMAISY50RXooWHVFXBC/rJzuWVyNEb3Q4TQkY+MpPr13Ynzqp3zUqgO Pg4jfT31ktuND9Ajoicb9VoLBcJe6GubHiSgM98cLjyBD4YOakeCvZQfXo+E9kRhfUNa U5ScH8nfRvOqx87oWDVrGDIhy7OIJbPP7PT3tC0jCPbd3QE4T/VGM2gsKwVHwf++P+rH b7Sw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AWQlbU8V; spf=pass (google.com: domain of y2kenny@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=y2kenny@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id b199sor3168169oih.93.2019.05.05.07.21.42 for (Google Transport Security); Sun, 05 May 2019 07:21:42 -0700 (PDT) Received-SPF: pass (google.com: domain of y2kenny@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AWQlbU8V; spf=pass (google.com: domain of y2kenny@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=y2kenny@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IQqSIA7uD5M0wQf7Hipk4HlzkvDnQtg0hg1zEHr33zY=; b=AWQlbU8VoVdVOq3fIViQl9WYLUDY46SLhaBmW9Msab5SMYgL0D5pHgqnoecO4JXrdp 2K/zjn5/HgnHS3DI4TKeICobLr5jNn3MZFF8AWGvOzgMQvaFxHj0UersObHSwzbqjCHU kXFFzOlrutozkLHx0JC/PEftOVW4BUW7jl9QNtEAmrYuDNdqb7lyNN17WYTdeLCmWnPy UnxDtMS+e3G9oMwruAOb/RYs10NL5OqNm6D+D7hRvzETt/45lB6Xuk6D+wbdv1aVOktJ iYhmorWgDXqeGGDkgoGwbfAGo4MEe7F2Ixq+NKElGpK8fo9OAlyHVml2Qc8WS9v2A1Yi 1sNg== X-Google-Smtp-Source: APXvYqxtCTlnFpaog3a+ume/+CFEM0o9HMexE6GdSPSB//kAtVjADF27eCZpP5vpBUt2GA/SlUqk8VsrOlucOhccchs= X-Received: by 2002:aca:72c9:: with SMTP id p192mr5372420oic.164.1557066102015; Sun, 05 May 2019 07:21:42 -0700 (PDT) MIME-Version: 1.0 References: <20190501140438.9506-1-brian.welty@intel.com> <20190502083433.GP7676@mtr-leonro.mtl.com> <20190505071436.GD6938@mtr-leonro.mtl.com> In-Reply-To: <20190505071436.GD6938@mtr-leonro.mtl.com> From: Kenny Ho Date: Sun, 5 May 2019 10:21:30 -0400 Message-ID: Subject: Re: [RFC PATCH 0/5] cgroup support for GPU devices To: Leon Romanovsky Cc: "Welty, Brian" , Alex Deucher , Parav Pandit , David Airlie , intel-gfx@lists.freedesktop.org, "J??r??me Glisse" , dri-devel@lists.freedesktop.org, Michal Hocko , linux-mm@kvack.org, Rodrigo Vivi , Li Zefan , Vladimir Davydov , Johannes Weiner , Tejun Heo , cgroups@vger.kernel.org, "Christian K??nig" , RDMA mailing list , kenny.ho@amd.com, Harish.Kasiviswanathan@amd.com, daniel@ffwll.ch Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, May 5, 2019 at 3:14 AM Leon Romanovsky wrote: > > > Doesn't RDMA already has a separate cgroup? Why not implement it there? > > > > > > > Hi Kenny, I can't answer for Leon, but I'm hopeful he agrees with rationale > > I gave in the cover letter. Namely, to implement in rdma controller, would > > mean duplicating existing memcg controls there. > > Exactly, I didn't feel comfortable to add notion of "device memory" > to RDMA cgroup and postponed that decision to later point of time. > RDMA operates with verbs objects and all our user space API is based around > that concept. At the end, system administrator will have hard time to > understand the differences between memcg and RDMA memory. Interesting. I actually don't understand this part (I worked in devops/sysadmin side of things but never with rdma.) Don't applications that use rdma require some awareness of rdma (I mean, you mentioned verbs and objects... or do they just use regular malloc for buffer allocation and then send it through some function?) As a user, I would have this question: why do I need to configure some part of rdma resources under rdma cgroup while other part of rdma resources in a different, seemingly unrelated cgroups. I think we need to be careful about drawing the line between duplication and over couplings between subsystems. I have other thoughts and concerns and I will try to organize them into a response in the next few days. Regards, Kenny > > > > Is AMD interested in collaborating to help shape this framework? > > It is intended to be device-neutral, so could be leveraged by various > > types of devices. > > If you have an alternative solution well underway, then maybe > > we can work together to merge our efforts into one. > > In the end, the DRM community is best served with common solution. > > > > > > > > > >>> and with future work, we could extend to: > > >>> * track and control share of GPU time (reuse of cpu/cpuacct) > > >>> * apply mask of allowed execution engines (reuse of cpusets) > > >>> > > >>> Instead of introducing a new cgroup subsystem for GPU devices, a new > > >>> framework is proposed to allow devices to register with existing cgroup > > >>> controllers, which creates per-device cgroup_subsys_state within the > > >>> cgroup. This gives device drivers their own private cgroup controls > > >>> (such as memory limits or other parameters) to be applied to device > > >>> resources instead of host system resources. > > >>> Device drivers (GPU or other) are then able to reuse the existing cgroup > > >>> controls, instead of inventing similar ones. > > >>> > > >>> Per-device controls would be exposed in cgroup filesystem as: > > >>> mount//.devices// > > >>> such as (for example): > > >>> mount//memory.devices//memory.max > > >>> mount//memory.devices//memory.current > > >>> mount//cpu.devices//cpu.stat > > >>> mount//cpu.devices//cpu.weight > > >>> > > >>> The drm/i915 patch in this series is based on top of other RFC work [1] > > >>> for i915 device memory support. > > >>> > > >>> AMD [2] and Intel [3] have proposed related work in this area within the > > >>> last few years, listed below as reference. This new RFC reuses existing > > >>> cgroup controllers and takes a different approach than prior work. > > >>> > > >>> Finally, some potential discussion points for this series: > > >>> * merge proposed .devices into a single devices directory? > > >>> * allow devices to have multiple registrations for subsets of resources? > > >>> * document a 'common charging policy' for device drivers to follow? > > >>> > > >>> [1] https://patchwork.freedesktop.org/series/56683/ > > >>> [2] https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html > > >>> [3] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html > > >>> > > >>> > > >>> Brian Welty (5): > > >>> cgroup: Add cgroup_subsys per-device registration framework > > >>> cgroup: Change kernfs_node for directories to store > > >>> cgroup_subsys_state > > >>> memcg: Add per-device support to memory cgroup subsystem > > >>> drm: Add memory cgroup registration and DRIVER_CGROUPS feature bit > > >>> drm/i915: Use memory cgroup for enforcing device memory limit > > >>> > > >>> drivers/gpu/drm/drm_drv.c | 12 + > > >>> drivers/gpu/drm/drm_gem.c | 7 + > > >>> drivers/gpu/drm/i915/i915_drv.c | 2 +- > > >>> drivers/gpu/drm/i915/intel_memory_region.c | 24 +- > > >>> include/drm/drm_device.h | 3 + > > >>> include/drm/drm_drv.h | 8 + > > >>> include/drm/drm_gem.h | 11 + > > >>> include/linux/cgroup-defs.h | 28 ++ > > >>> include/linux/cgroup.h | 3 + > > >>> include/linux/memcontrol.h | 10 + > > >>> kernel/cgroup/cgroup-v1.c | 10 +- > > >>> kernel/cgroup/cgroup.c | 310 ++++++++++++++++++--- > > >>> mm/memcontrol.c | 183 +++++++++++- > > >>> 13 files changed, 552 insertions(+), 59 deletions(-) > > >>> > > >>> -- > > >>> 2.21.0 > > >>> > > >> _______________________________________________ > > >> dri-devel mailing list > > >> dri-devel@lists.freedesktop.org > > >> https://lists.freedesktop.org/mailman/listinfo/dri-devel