From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.1 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FEC7C33CB7 for ; Sun, 19 Jan 2020 13:04:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A2F2620679 for ; Sun, 19 Jan 2020 13:04:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="iWTCYkYe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A2F2620679 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 282A66B0576; Sun, 19 Jan 2020 08:04:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 233986B0577; Sun, 19 Jan 2020 08:04:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FA046B0578; Sun, 19 Jan 2020 08:04:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0054.hostedemail.com [216.40.44.54]) by kanga.kvack.org (Postfix) with ESMTP id E1AFB6B0576 for ; Sun, 19 Jan 2020 08:04:09 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6FA70442B for ; Sun, 19 Jan 2020 13:04:09 +0000 (UTC) X-FDA: 76394401818.01.joke89_7098efa98022 X-HE-Tag: joke89_7098efa98022 X-Filterd-Recvd-Size: 15328 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2066.outbound.protection.outlook.com [40.107.92.66]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Sun, 19 Jan 2020 13:04:08 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TSmms3gtm/z/her+7dIUuhdsvERRp+SocQMHmMUeUfXoZfnrgFS7onSz3+Utps4QXkPWiVd4u8y1f0L11F+6YV92YqjQJkV/uSPzIiwz+meySTqSuBukObhx2OnEDbiQxTuw2iC0WFpNr8/30foqGK81xNFqLt1uRakwiyl0+J6XvI+8GElbkXdhOuhE5lV2TaoZ6jffo4n7yRNwhP45kH7np5NnSrXdcqvlYFw745+ano9EnCwILkkEJkZtaVYJ8RYMK80h3WMXMNTyKJKtryGRthncaZOpgjIXTkrnvTQuk/RT9VXXoy40HYmG6YowlI59WUvTVirNqF5L3Ls38A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Lc3KPrSxN4ypxUGuaIhLzwAb/+yrCobaCHQgDGRk9Q=; b=DWHf0VuQRO+x3TPdpg/JA8kT8/LtHSZ65OjwPhGm7yr2kX94ASfJ85dVUbjECo1JKanWFE+oi/rPFvyFStFqHW3mZ+OT2lthPcopjRCF0vfj8WIRbs3KKGPwNUl7mjWb4I/8bso2aCE0wUak8g/ZLEHdf9BOkHcKvUyyxja3O60bWd0S4okrsKu+SLSbxdwKMKvEHtZQ+xVd/DtDarTvdmjfbwDULP0016xoeqDPOhYtEUZyt6U3jIKK3MwkT+HxucN5u6SxXg9ObwVpQmbWOaYKQwRj3OgQxwZRFPiWoVVzo/g1jVsUDvcZcaMg9Y0Y9xEi7WHbApfwQIyvtHqKeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Lc3KPrSxN4ypxUGuaIhLzwAb/+yrCobaCHQgDGRk9Q=; b=iWTCYkYeY7NTtgfDfHmx4h66voULX2xbgR06XKj/F0tgzopXvax2aWlCfEuYAGwKNR/mKmJzVVFS8QvbjfEjtneF4zDzuP0mzEfrpFusRQZeNd/Yh8faEggDGYTeM0CUnS8VUzasAYQlVASKHlhMq0NiIMUgWYP/XbS1Zx46RKQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Christian.Koenig@amd.com; Received: from DM5PR12MB1705.namprd12.prod.outlook.com (10.175.88.22) by DM5PR12MB1177.namprd12.prod.outlook.com (10.168.233.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2644.19; Sun, 19 Jan 2020 13:04:05 +0000 Received: from DM5PR12MB1705.namprd12.prod.outlook.com ([fe80::8dde:b52a:d97a:e89]) by DM5PR12MB1705.namprd12.prod.outlook.com ([fe80::8dde:b52a:d97a:e89%2]) with mapi id 15.20.2644.024; Sun, 19 Jan 2020 13:04:05 +0000 Subject: Re: [PATCH RFC 3/3] drm/ttm: support memcg for ttm_tt To: Qiang Yu Cc: Qiang Yu , Linux Memory Management List , cgroups@vger.kernel.org, dri-devel , David Airlie , Kenny Ho , Michal Hocko , Huang Rui , Johannes Weiner , Tejun Heo , Andrew Morton References: <20200113153543.24957-1-qiang.yu@amd.com> <20200113153543.24957-4-qiang.yu@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: <0164fcbf-393a-237f-69d7-aa26dabe6ad9@amd.com> Date: Sun, 19 Jan 2020 14:03:58 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-ClientProxiedBy: AM0PR06CA0069.eurprd06.prod.outlook.com (2603:10a6:208:aa::46) To DM5PR12MB1705.namprd12.prod.outlook.com (2603:10b6:3:10c::22) MIME-Version: 1.0 Received: from [IPv6:2a02:908:1252:fb60:be8a:bd56:1f94:86e7] (2a02:908:1252:fb60:be8a:bd56:1f94:86e7) by AM0PR06CA0069.eurprd06.prod.outlook.com (2603:10a6:208:aa::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2644.20 via Frontend Transport; Sun, 19 Jan 2020 13:04:03 +0000 X-Originating-IP: [2a02:908:1252:fb60:be8a:bd56:1f94:86e7] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 4167b2c0-f824-4390-52c3-08d79ce01009 X-MS-TrafficTypeDiagnostic: DM5PR12MB1177:|DM5PR12MB1177: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-Forefront-PRVS: 0287BBA78D X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4636009)(39860400002)(366004)(396003)(136003)(376002)(346002)(199004)(189003)(316002)(4326008)(31696002)(5660300002)(86362001)(45080400002)(36756003)(6486002)(8936002)(66946007)(8676002)(81156014)(54906003)(2616005)(66556008)(66476007)(6916009)(81166006)(52116002)(66574012)(53546011)(2906002)(478600001)(966005)(186003)(31686004)(6666004)(16526019);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR12MB1177;H:DM5PR12MB1705.namprd12.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6Hw3Z0mPHU48t+e0oEHQRfrWcxqBHQQkveTtRQqImAMXA0qUM9I8EJEcNxSQL0eMbtYRRI3pU4wz71BCn711SZpnfV93EYzaB5/dJuYIbJ41McZHDNvO7KCBweK3BzHdCAJONV15g5gmnXrGP1M5KZw9Fif4CCXyMXTXnMNVLN59OVlkuOXdQ+50SC2BgytCrAvczs8ctcvRFNgLMMlkZ/TkWjWgiE5A8rvHwYYVVUQEL5t5bIFrLuO0vsFrfWqv2C6oJ3USgqt/HZaqRKWTQ26O/3rNyUYZJ7NpwM9dcUbxC7PQAk3dis2SaGprG6aCKic0qgCEwwBze3UmzQXL2ZD42VTScQ0Om4BMbPbjrOWsrUlh27PQQ4G+ZRWCnKU9BMiOuaVwAtTkrZiz8HMoawrXh+QSha2uHRBlffZKbYb6SfWbJjhVzhoZqlmJgDJ5nkNMZuyqcm7vzHqEQTYJVXi8sF2alYpwH1qbwYHTcbw= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4167b2c0-f824-4390-52c3-08d79ce01009 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jan 2020 13:04:05.5426 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CvhTWEGuBBJ1WWCfeoFj+Jwao5QwMFo4aijXaVUrADrKzOJAwzaLs8mzV15dCqwU X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1177 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Am 19.01.20 um 03:47 schrieb Qiang Yu: > On Mon, Jan 13, 2020 at 11:56 PM Christian K=C3=B6nig > wrote: >> Am 13.01.20 um 16:35 schrieb Qiang Yu: >>> Charge TTM allocated system memory to memory cgroup which will >>> limit the memory usage of a group of processes. >> NAK to the whole approach. This belongs into the GEM or driver layer, >> but not into TTM. >> > Sorry for responding late. > > GEM layer seems not a proper place to handle this as: > 1. it is not aware of the back storage (system mem or device mem) unles= s > we add this information up to GEM which I think is not appropriate > 2. system memory allocated by GEM with drm_gem_get_pages() is already > charged to memcg, it's only the ttm system memory not charged to memcg The key point is that we already discussed this on the mailing list and=20 GEM was agreed on to be the right place for this. That's the reason why the Intel developers already proposed a way to=20 expose the buffer location in GEM. Please sync up with Kenny who is leading the development efforts and=20 with the Intel developers before warming up an old discussion again. Adding that to TTM is an absolute no-go from my maintainers perspective. > > Implement in driver like amdgpu is an option. But seems the problem is = inside > TTM which does not charge pages allocated by itself to memcg, won't it = be > better to solve it in TTM so that all drivers using it can benefit? Or = you just > think we should not rely on memcg for GPU system memory limitation? > >>> The memory is always charged to the control group of task which >>> create this buffer object and when it's created. For example, >>> when a buffer is created by process A and exported to process B, >>> then process B populate this buffer, the memory is still charged >>> to process A's memcg; if a buffer is created by process A when in >>> memcg B, then A is moved to memcg C and populate this buffer, it >>> will charge memcg B. >> This is actually the most common use case for graphics application whe= re >> the X server allocates most of the backing store. >> >> So we need a better handling than just accounting the memory to whoeve= r >> allocated it first. >> > You mean the application based on DRI2 and X11 protocol draw? I think t= his > is still reasonable to charge xserver for the memory, because xserver a= llocate > the buffer and share to application which is its design and implementat= ion > nature. With DRI3, the buffer is allocated by application, also > suitable for this > approach. That is a way to simplistic. Again we already discussed this and the agreed compromise is to charge=20 the application which is using the memory and not who has allocated it. So you need to add the charge on importing a buffer and not just when it=20 is created. Regards, Christian. > > Regards, > Qiang > >> Regards, >> Christian. >> >>> Signed-off-by: Qiang Yu >>> --- >>> drivers/gpu/drm/ttm/ttm_bo.c | 10 ++++++++++ >>> drivers/gpu/drm/ttm/ttm_page_alloc.c | 18 +++++++++++++++++- >>> drivers/gpu/drm/ttm/ttm_tt.c | 3 +++ >>> include/drm/ttm/ttm_bo_api.h | 5 +++++ >>> include/drm/ttm/ttm_tt.h | 4 ++++ >>> 5 files changed, 39 insertions(+), 1 deletion(-) >>> >>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_b= o.c >>> index 8d91b0428af1..4e64846ee523 100644 >>> --- a/drivers/gpu/drm/ttm/ttm_bo.c >>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c >>> @@ -42,6 +42,7 @@ >>> #include >>> #include >>> #include >>> +#include >>> >>> static void ttm_bo_global_kobj_release(struct kobject *kobj); >>> >>> @@ -162,6 +163,10 @@ static void ttm_bo_release_list(struct kref *lis= t_kref) >>> if (!ttm_bo_uses_embedded_gem_object(bo)) >>> dma_resv_fini(&bo->base._resv); >>> mutex_destroy(&bo->wu_mutex); >>> +#ifdef CONFIG_MEMCG >>> + if (bo->memcg) >>> + css_put(&bo->memcg->css); >>> +#endif >>> bo->destroy(bo); >>> ttm_mem_global_free(&ttm_mem_glob, acc_size); >>> } >>> @@ -1330,6 +1335,11 @@ int ttm_bo_init_reserved(struct ttm_bo_device = *bdev, >>> } >>> atomic_inc(&ttm_bo_glob.bo_count); >>> >>> +#ifdef CONFIG_MEMCG >>> + if (bo->type =3D=3D ttm_bo_type_device) >>> + bo->memcg =3D mem_cgroup_driver_get_from_current(); >>> +#endif >>> + >>> /* >>> * For ttm_bo_type_device buffers, allocate >>> * address space from the device. >>> diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc.c b/drivers/gpu/drm/t= tm/ttm_page_alloc.c >>> index b40a4678c296..ecd1831a1d38 100644 >>> --- a/drivers/gpu/drm/ttm/ttm_page_alloc.c >>> +++ b/drivers/gpu/drm/ttm/ttm_page_alloc.c >>> @@ -42,7 +42,7 @@ >>> #include /* for seq_printf */ >>> #include >>> #include >>> - >>> +#include >>> #include >>> >>> #include >>> @@ -1045,6 +1045,11 @@ ttm_pool_unpopulate_helper(struct ttm_tt *ttm,= unsigned mem_count_update) >>> ttm_put_pages(ttm->pages, ttm->num_pages, ttm->page_flags, >>> ttm->caching_state); >>> ttm->state =3D tt_unpopulated; >>> + >>> +#ifdef CONFIG_MEMCG >>> + if (ttm->memcg) >>> + mem_cgroup_uncharge_drvmem(ttm->memcg, ttm->num_pages); >>> +#endif >>> } >>> >>> int ttm_pool_populate(struct ttm_tt *ttm, struct ttm_operation_ctx= *ctx) >>> @@ -1059,6 +1064,17 @@ int ttm_pool_populate(struct ttm_tt *ttm, stru= ct ttm_operation_ctx *ctx) >>> if (ttm_check_under_lowerlimit(mem_glob, ttm->num_pages, ctx)) >>> return -ENOMEM; >>> >>> +#ifdef CONFIG_MEMCG >>> + if (ttm->memcg) { >>> + gfp_t gfp_flags =3D GFP_USER; >>> + if (ttm->page_flags & TTM_PAGE_FLAG_NO_RETRY) >>> + gfp_flags |=3D __GFP_RETRY_MAYFAIL; >>> + ret =3D mem_cgroup_charge_drvmem(ttm->memcg, gfp_flags,= ttm->num_pages); >>> + if (ret) >>> + return ret; >>> + } >>> +#endif >>> + >>> ret =3D ttm_get_pages(ttm->pages, ttm->num_pages, ttm->page_fl= ags, >>> ttm->caching_state); >>> if (unlikely(ret !=3D 0)) { >>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_t= t.c >>> index e0e9b4f69db6..1acb153084e1 100644 >>> --- a/drivers/gpu/drm/ttm/ttm_tt.c >>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c >>> @@ -233,6 +233,9 @@ void ttm_tt_init_fields(struct ttm_tt *ttm, struc= t ttm_buffer_object *bo, >>> ttm->state =3D tt_unpopulated; >>> ttm->swap_storage =3D NULL; >>> ttm->sg =3D bo->sg; >>> +#ifdef CONFIG_MEMCG >>> + ttm->memcg =3D bo->memcg; >>> +#endif >>> } >>> >>> int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo, >>> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_ap= i.h >>> index 65e399d280f7..95a08e81a73e 100644 >>> --- a/include/drm/ttm/ttm_bo_api.h >>> +++ b/include/drm/ttm/ttm_bo_api.h >>> @@ -54,6 +54,8 @@ struct ttm_place; >>> >>> struct ttm_lru_bulk_move; >>> >>> +struct mem_cgroup; >>> + >>> /** >>> * struct ttm_bus_placement >>> * >>> @@ -180,6 +182,9 @@ struct ttm_buffer_object { >>> void (*destroy) (struct ttm_buffer_object *); >>> unsigned long num_pages; >>> size_t acc_size; >>> +#ifdef CONFIG_MEMCG >>> + struct mem_cgroup *memcg; >>> +#endif >>> >>> /** >>> * Members not needing protection. >>> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h >>> index c0e928abf592..10fb5a557b95 100644 >>> --- a/include/drm/ttm/ttm_tt.h >>> +++ b/include/drm/ttm/ttm_tt.h >>> @@ -33,6 +33,7 @@ struct ttm_tt; >>> struct ttm_mem_reg; >>> struct ttm_buffer_object; >>> struct ttm_operation_ctx; >>> +struct mem_cgroup; >>> >>> #define TTM_PAGE_FLAG_WRITE (1 << 3) >>> #define TTM_PAGE_FLAG_SWAPPED (1 << 4) >>> @@ -116,6 +117,9 @@ struct ttm_tt { >>> tt_unbound, >>> tt_unpopulated, >>> } state; >>> +#ifdef CONFIG_MEMCG >>> + struct mem_cgroup *memcg; >>> +#endif >>> }; >>> >>> /** >> _______________________________________________ >> dri-devel mailing list >> dri-devel@lists.freedesktop.org >> https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fli= sts.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&data=3D02%7C01%7= Cchristian.koenig%40amd.com%7C5d3b70a43b80444c550808d79c89e968%7C3dd8961f= e4884e608e11a82d994e183d%7C0%7C0%7C637149988466853762&sdata=3Dni3ku7n= C%2FD5E8kivppfuuF7ZoiyfLQ8L3Y4j9IfHYUU%3D&reserved=3D0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.1 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7E7FC33CB1 for ; Sun, 19 Jan 2020 13:04:11 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B06AD2073A for ; Sun, 19 Jan 2020 13:04:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="iWTCYkYe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B06AD2073A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B355E6E02D; Sun, 19 Jan 2020 13:04:09 +0000 (UTC) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2085.outbound.protection.outlook.com [40.107.92.85]) by gabe.freedesktop.org (Postfix) with ESMTPS id 996E36E02D for ; Sun, 19 Jan 2020 13:04:08 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TSmms3gtm/z/her+7dIUuhdsvERRp+SocQMHmMUeUfXoZfnrgFS7onSz3+Utps4QXkPWiVd4u8y1f0L11F+6YV92YqjQJkV/uSPzIiwz+meySTqSuBukObhx2OnEDbiQxTuw2iC0WFpNr8/30foqGK81xNFqLt1uRakwiyl0+J6XvI+8GElbkXdhOuhE5lV2TaoZ6jffo4n7yRNwhP45kH7np5NnSrXdcqvlYFw745+ano9EnCwILkkEJkZtaVYJ8RYMK80h3WMXMNTyKJKtryGRthncaZOpgjIXTkrnvTQuk/RT9VXXoy40HYmG6YowlI59WUvTVirNqF5L3Ls38A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Lc3KPrSxN4ypxUGuaIhLzwAb/+yrCobaCHQgDGRk9Q=; b=DWHf0VuQRO+x3TPdpg/JA8kT8/LtHSZ65OjwPhGm7yr2kX94ASfJ85dVUbjECo1JKanWFE+oi/rPFvyFStFqHW3mZ+OT2lthPcopjRCF0vfj8WIRbs3KKGPwNUl7mjWb4I/8bso2aCE0wUak8g/ZLEHdf9BOkHcKvUyyxja3O60bWd0S4okrsKu+SLSbxdwKMKvEHtZQ+xVd/DtDarTvdmjfbwDULP0016xoeqDPOhYtEUZyt6U3jIKK3MwkT+HxucN5u6SxXg9ObwVpQmbWOaYKQwRj3OgQxwZRFPiWoVVzo/g1jVsUDvcZcaMg9Y0Y9xEi7WHbApfwQIyvtHqKeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Lc3KPrSxN4ypxUGuaIhLzwAb/+yrCobaCHQgDGRk9Q=; b=iWTCYkYeY7NTtgfDfHmx4h66voULX2xbgR06XKj/F0tgzopXvax2aWlCfEuYAGwKNR/mKmJzVVFS8QvbjfEjtneF4zDzuP0mzEfrpFusRQZeNd/Yh8faEggDGYTeM0CUnS8VUzasAYQlVASKHlhMq0NiIMUgWYP/XbS1Zx46RKQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Christian.Koenig@amd.com; Received: from DM5PR12MB1705.namprd12.prod.outlook.com (10.175.88.22) by DM5PR12MB1177.namprd12.prod.outlook.com (10.168.233.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2644.19; Sun, 19 Jan 2020 13:04:05 +0000 Received: from DM5PR12MB1705.namprd12.prod.outlook.com ([fe80::8dde:b52a:d97a:e89]) by DM5PR12MB1705.namprd12.prod.outlook.com ([fe80::8dde:b52a:d97a:e89%2]) with mapi id 15.20.2644.024; Sun, 19 Jan 2020 13:04:05 +0000 Subject: Re: [PATCH RFC 3/3] drm/ttm: support memcg for ttm_tt To: Qiang Yu References: <20200113153543.24957-1-qiang.yu@amd.com> <20200113153543.24957-4-qiang.yu@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: <0164fcbf-393a-237f-69d7-aa26dabe6ad9@amd.com> Date: Sun, 19 Jan 2020 14:03:58 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 In-Reply-To: Content-Language: en-US X-ClientProxiedBy: AM0PR06CA0069.eurprd06.prod.outlook.com (2603:10a6:208:aa::46) To DM5PR12MB1705.namprd12.prod.outlook.com (2603:10b6:3:10c::22) MIME-Version: 1.0 Received: from [IPv6:2a02:908:1252:fb60:be8a:bd56:1f94:86e7] (2a02:908:1252:fb60:be8a:bd56:1f94:86e7) by AM0PR06CA0069.eurprd06.prod.outlook.com (2603:10a6:208:aa::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2644.20 via Frontend Transport; Sun, 19 Jan 2020 13:04:03 +0000 X-Originating-IP: [2a02:908:1252:fb60:be8a:bd56:1f94:86e7] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 4167b2c0-f824-4390-52c3-08d79ce01009 X-MS-TrafficTypeDiagnostic: DM5PR12MB1177:|DM5PR12MB1177: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-Forefront-PRVS: 0287BBA78D X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(366004)(396003)(136003)(376002)(346002)(199004)(189003)(316002)(4326008)(31696002)(5660300002)(86362001)(45080400002)(36756003)(6486002)(8936002)(66946007)(8676002)(81156014)(54906003)(2616005)(66556008)(66476007)(6916009)(81166006)(52116002)(66574012)(53546011)(2906002)(478600001)(966005)(186003)(31686004)(6666004)(16526019); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR12MB1177; H:DM5PR12MB1705.namprd12.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6Hw3Z0mPHU48t+e0oEHQRfrWcxqBHQQkveTtRQqImAMXA0qUM9I8EJEcNxSQL0eMbtYRRI3pU4wz71BCn711SZpnfV93EYzaB5/dJuYIbJ41McZHDNvO7KCBweK3BzHdCAJONV15g5gmnXrGP1M5KZw9Fif4CCXyMXTXnMNVLN59OVlkuOXdQ+50SC2BgytCrAvczs8ctcvRFNgLMMlkZ/TkWjWgiE5A8rvHwYYVVUQEL5t5bIFrLuO0vsFrfWqv2C6oJ3USgqt/HZaqRKWTQ26O/3rNyUYZJ7NpwM9dcUbxC7PQAk3dis2SaGprG6aCKic0qgCEwwBze3UmzQXL2ZD42VTScQ0Om4BMbPbjrOWsrUlh27PQQ4G+ZRWCnKU9BMiOuaVwAtTkrZiz8HMoawrXh+QSha2uHRBlffZKbYb6SfWbJjhVzhoZqlmJgDJ5nkNMZuyqcm7vzHqEQTYJVXi8sF2alYpwH1qbwYHTcbw= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4167b2c0-f824-4390-52c3-08d79ce01009 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jan 2020 13:04:05.5426 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CvhTWEGuBBJ1WWCfeoFj+Jwao5QwMFo4aijXaVUrADrKzOJAwzaLs8mzV15dCqwU X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1177 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Linux Memory Management List , David Airlie , Kenny Ho , dri-devel , Michal Hocko , Qiang Yu , Huang Rui , Johannes Weiner , Tejun Heo , cgroups@vger.kernel.org, Andrew Morton Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" QW0gMTkuMDEuMjAgdW0gMDM6NDcgc2NocmllYiBRaWFuZyBZdToKPiBPbiBNb24sIEphbiAxMywg MjAyMCBhdCAxMTo1NiBQTSBDaHJpc3RpYW4gS8O2bmlnCj4gPGNocmlzdGlhbi5rb2VuaWdAYW1k LmNvbT4gd3JvdGU6Cj4+IEFtIDEzLjAxLjIwIHVtIDE2OjM1IHNjaHJpZWIgUWlhbmcgWXU6Cj4+ PiBDaGFyZ2UgVFRNIGFsbG9jYXRlZCBzeXN0ZW0gbWVtb3J5IHRvIG1lbW9yeSBjZ3JvdXAgd2hp Y2ggd2lsbAo+Pj4gbGltaXQgdGhlIG1lbW9yeSB1c2FnZSBvZiBhIGdyb3VwIG9mIHByb2Nlc3Nl cy4KPj4gTkFLIHRvIHRoZSB3aG9sZSBhcHByb2FjaC4gVGhpcyBiZWxvbmdzIGludG8gdGhlIEdF TSBvciBkcml2ZXIgbGF5ZXIsCj4+IGJ1dCBub3QgaW50byBUVE0uCj4+Cj4gU29ycnkgZm9yIHJl c3BvbmRpbmcgbGF0ZS4KPgo+IEdFTSBsYXllciBzZWVtcyBub3QgYSBwcm9wZXIgcGxhY2UgdG8g aGFuZGxlIHRoaXMgYXM6Cj4gMS4gaXQgaXMgbm90IGF3YXJlIG9mIHRoZSBiYWNrIHN0b3JhZ2Ug KHN5c3RlbSBtZW0gb3IgZGV2aWNlIG1lbSkgdW5sZXNzCj4gd2UgYWRkIHRoaXMgaW5mb3JtYXRp b24gdXAgdG8gR0VNIHdoaWNoIEkgdGhpbmsgaXMgbm90IGFwcHJvcHJpYXRlCj4gMi4gc3lzdGVt IG1lbW9yeSBhbGxvY2F0ZWQgYnkgR0VNIHdpdGggZHJtX2dlbV9nZXRfcGFnZXMoKSBpcyBhbHJl YWR5Cj4gY2hhcmdlZCB0byBtZW1jZywgaXQncyBvbmx5IHRoZSB0dG0gc3lzdGVtIG1lbW9yeSBu b3QgY2hhcmdlZCB0byBtZW1jZwoKVGhlIGtleSBwb2ludCBpcyB0aGF0IHdlIGFscmVhZHkgZGlz Y3Vzc2VkIHRoaXMgb24gdGhlIG1haWxpbmcgbGlzdCBhbmQgCkdFTSB3YXMgYWdyZWVkIG9uIHRv IGJlIHRoZSByaWdodCBwbGFjZSBmb3IgdGhpcy4KClRoYXQncyB0aGUgcmVhc29uIHdoeSB0aGUg SW50ZWwgZGV2ZWxvcGVycyBhbHJlYWR5IHByb3Bvc2VkIGEgd2F5IHRvIApleHBvc2UgdGhlIGJ1 ZmZlciBsb2NhdGlvbiBpbiBHRU0uCgpQbGVhc2Ugc3luYyB1cCB3aXRoIEtlbm55IHdobyBpcyBs ZWFkaW5nIHRoZSBkZXZlbG9wbWVudCBlZmZvcnRzIGFuZCAKd2l0aCB0aGUgSW50ZWwgZGV2ZWxv cGVycyBiZWZvcmUgd2FybWluZyB1cCBhbiBvbGQgZGlzY3Vzc2lvbiBhZ2Fpbi4KCkFkZGluZyB0 aGF0IHRvIFRUTSBpcyBhbiBhYnNvbHV0ZSBuby1nbyBmcm9tIG15IG1haW50YWluZXJzIHBlcnNw ZWN0aXZlLgoKPgo+IEltcGxlbWVudCBpbiBkcml2ZXIgbGlrZSBhbWRncHUgaXMgYW4gb3B0aW9u LiBCdXQgc2VlbXMgdGhlIHByb2JsZW0gaXMgaW5zaWRlCj4gVFRNIHdoaWNoIGRvZXMgbm90IGNo YXJnZSBwYWdlcyBhbGxvY2F0ZWQgYnkgaXRzZWxmIHRvIG1lbWNnLCB3b24ndCBpdCBiZQo+IGJl dHRlciB0byBzb2x2ZSBpdCBpbiBUVE0gc28gdGhhdCBhbGwgZHJpdmVycyB1c2luZyBpdCBjYW4g YmVuZWZpdD8gT3IgeW91IGp1c3QKPiB0aGluayB3ZSBzaG91bGQgbm90IHJlbHkgb24gbWVtY2cg Zm9yIEdQVSBzeXN0ZW0gbWVtb3J5IGxpbWl0YXRpb24/Cj4KPj4+IFRoZSBtZW1vcnkgaXMgYWx3 YXlzIGNoYXJnZWQgdG8gdGhlIGNvbnRyb2wgZ3JvdXAgb2YgdGFzayB3aGljaAo+Pj4gY3JlYXRl IHRoaXMgYnVmZmVyIG9iamVjdCBhbmQgd2hlbiBpdCdzIGNyZWF0ZWQuIEZvciBleGFtcGxlLAo+ Pj4gd2hlbiBhIGJ1ZmZlciBpcyBjcmVhdGVkIGJ5IHByb2Nlc3MgQSBhbmQgZXhwb3J0ZWQgdG8g cHJvY2VzcyBCLAo+Pj4gdGhlbiBwcm9jZXNzIEIgcG9wdWxhdGUgdGhpcyBidWZmZXIsIHRoZSBt ZW1vcnkgaXMgc3RpbGwgY2hhcmdlZAo+Pj4gdG8gcHJvY2VzcyBBJ3MgbWVtY2c7IGlmIGEgYnVm ZmVyIGlzIGNyZWF0ZWQgYnkgcHJvY2VzcyBBIHdoZW4gaW4KPj4+IG1lbWNnIEIsIHRoZW4gQSBp cyBtb3ZlZCB0byBtZW1jZyBDIGFuZCBwb3B1bGF0ZSB0aGlzIGJ1ZmZlciwgaXQKPj4+IHdpbGwg Y2hhcmdlIG1lbWNnIEIuCj4+IFRoaXMgaXMgYWN0dWFsbHkgdGhlIG1vc3QgY29tbW9uIHVzZSBj YXNlIGZvciBncmFwaGljcyBhcHBsaWNhdGlvbiB3aGVyZQo+PiB0aGUgWCBzZXJ2ZXIgYWxsb2Nh dGVzIG1vc3Qgb2YgdGhlIGJhY2tpbmcgc3RvcmUuCj4+Cj4+IFNvIHdlIG5lZWQgYSBiZXR0ZXIg aGFuZGxpbmcgdGhhbiBqdXN0IGFjY291bnRpbmcgdGhlIG1lbW9yeSB0byB3aG9ldmVyCj4+IGFs bG9jYXRlZCBpdCBmaXJzdC4KPj4KPiBZb3UgbWVhbiB0aGUgYXBwbGljYXRpb24gYmFzZWQgb24g RFJJMiBhbmQgWDExIHByb3RvY29sIGRyYXc/IEkgdGhpbmsgdGhpcwo+IGlzIHN0aWxsIHJlYXNv bmFibGUgdG8gY2hhcmdlIHhzZXJ2ZXIgZm9yIHRoZSBtZW1vcnksIGJlY2F1c2UgeHNlcnZlciBh bGxvY2F0ZQo+IHRoZSBidWZmZXIgYW5kIHNoYXJlIHRvIGFwcGxpY2F0aW9uIHdoaWNoIGlzIGl0 cyBkZXNpZ24gYW5kIGltcGxlbWVudGF0aW9uCj4gbmF0dXJlLiBXaXRoIERSSTMsIHRoZSBidWZm ZXIgaXMgYWxsb2NhdGVkIGJ5IGFwcGxpY2F0aW9uLCBhbHNvCj4gc3VpdGFibGUgZm9yIHRoaXMK PiBhcHByb2FjaC4KClRoYXQgaXMgYSB3YXkgdG8gc2ltcGxpc3RpYy4KCkFnYWluIHdlIGFscmVh ZHkgZGlzY3Vzc2VkIHRoaXMgYW5kIHRoZSBhZ3JlZWQgY29tcHJvbWlzZSBpcyB0byBjaGFyZ2Ug CnRoZSBhcHBsaWNhdGlvbiB3aGljaCBpcyB1c2luZyB0aGUgbWVtb3J5IGFuZCBub3Qgd2hvIGhh cyBhbGxvY2F0ZWQgaXQuCgpTbyB5b3UgbmVlZCB0byBhZGQgdGhlIGNoYXJnZSBvbiBpbXBvcnRp bmcgYSBidWZmZXIgYW5kIG5vdCBqdXN0IHdoZW4gaXQgCmlzIGNyZWF0ZWQuCgpSZWdhcmRzLApD aHJpc3RpYW4uCgo+Cj4gUmVnYXJkcywKPiBRaWFuZwo+Cj4+IFJlZ2FyZHMsCj4+IENocmlzdGlh bi4KPj4KPj4+IFNpZ25lZC1vZmYtYnk6IFFpYW5nIFl1IDxxaWFuZy55dUBhbWQuY29tPgo+Pj4g LS0tCj4+PiAgICBkcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9iby5jICAgICAgICAgfCAxMCArKysr KysrKysrCj4+PiAgICBkcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9wYWdlX2FsbG9jLmMgfCAxOCAr KysrKysrKysrKysrKysrKy0KPj4+ICAgIGRyaXZlcnMvZ3B1L2RybS90dG0vdHRtX3R0LmMgICAg ICAgICB8ICAzICsrKwo+Pj4gICAgaW5jbHVkZS9kcm0vdHRtL3R0bV9ib19hcGkuaCAgICAgICAg IHwgIDUgKysrKysKPj4+ICAgIGluY2x1ZGUvZHJtL3R0bS90dG1fdHQuaCAgICAgICAgICAgICB8 ICA0ICsrKysKPj4+ICAgIDUgZmlsZXMgY2hhbmdlZCwgMzkgaW5zZXJ0aW9ucygrKSwgMSBkZWxl dGlvbigtKQo+Pj4KPj4+IGRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9iby5j IGIvZHJpdmVycy9ncHUvZHJtL3R0bS90dG1fYm8uYwo+Pj4gaW5kZXggOGQ5MWIwNDI4YWYxLi40 ZTY0ODQ2ZWU1MjMgMTAwNjQ0Cj4+PiAtLS0gYS9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9iby5j Cj4+PiArKysgYi9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9iby5jCj4+PiBAQCAtNDIsNiArNDIs NyBAQAo+Pj4gICAgI2luY2x1ZGUgPGxpbnV4L21vZHVsZS5oPgo+Pj4gICAgI2luY2x1ZGUgPGxp bnV4L2F0b21pYy5oPgo+Pj4gICAgI2luY2x1ZGUgPGxpbnV4L2RtYS1yZXN2Lmg+Cj4+PiArI2lu Y2x1ZGUgPGxpbnV4L21lbWNvbnRyb2wuaD4KPj4+Cj4+PiAgICBzdGF0aWMgdm9pZCB0dG1fYm9f Z2xvYmFsX2tvYmpfcmVsZWFzZShzdHJ1Y3Qga29iamVjdCAqa29iaik7Cj4+Pgo+Pj4gQEAgLTE2 Miw2ICsxNjMsMTAgQEAgc3RhdGljIHZvaWQgdHRtX2JvX3JlbGVhc2VfbGlzdChzdHJ1Y3Qga3Jl ZiAqbGlzdF9rcmVmKQo+Pj4gICAgICAgIGlmICghdHRtX2JvX3VzZXNfZW1iZWRkZWRfZ2VtX29i amVjdChibykpCj4+PiAgICAgICAgICAgICAgICBkbWFfcmVzdl9maW5pKCZiby0+YmFzZS5fcmVz dik7Cj4+PiAgICAgICAgbXV0ZXhfZGVzdHJveSgmYm8tPnd1X211dGV4KTsKPj4+ICsjaWZkZWYg Q09ORklHX01FTUNHCj4+PiArICAgICBpZiAoYm8tPm1lbWNnKQo+Pj4gKyAgICAgICAgICAgICBj c3NfcHV0KCZiby0+bWVtY2ctPmNzcyk7Cj4+PiArI2VuZGlmCj4+PiAgICAgICAgYm8tPmRlc3Ry b3koYm8pOwo+Pj4gICAgICAgIHR0bV9tZW1fZ2xvYmFsX2ZyZWUoJnR0bV9tZW1fZ2xvYiwgYWNj X3NpemUpOwo+Pj4gICAgfQo+Pj4gQEAgLTEzMzAsNiArMTMzNSwxMSBAQCBpbnQgdHRtX2JvX2lu aXRfcmVzZXJ2ZWQoc3RydWN0IHR0bV9ib19kZXZpY2UgKmJkZXYsCj4+PiAgICAgICAgfQo+Pj4g ICAgICAgIGF0b21pY19pbmMoJnR0bV9ib19nbG9iLmJvX2NvdW50KTsKPj4+Cj4+PiArI2lmZGVm IENPTkZJR19NRU1DRwo+Pj4gKyAgICAgaWYgKGJvLT50eXBlID09IHR0bV9ib190eXBlX2Rldmlj ZSkKPj4+ICsgICAgICAgICAgICAgYm8tPm1lbWNnID0gbWVtX2Nncm91cF9kcml2ZXJfZ2V0X2Zy b21fY3VycmVudCgpOwo+Pj4gKyNlbmRpZgo+Pj4gKwo+Pj4gICAgICAgIC8qCj4+PiAgICAgICAg ICogRm9yIHR0bV9ib190eXBlX2RldmljZSBidWZmZXJzLCBhbGxvY2F0ZQo+Pj4gICAgICAgICAq IGFkZHJlc3Mgc3BhY2UgZnJvbSB0aGUgZGV2aWNlLgo+Pj4gZGlmZiAtLWdpdCBhL2RyaXZlcnMv Z3B1L2RybS90dG0vdHRtX3BhZ2VfYWxsb2MuYyBiL2RyaXZlcnMvZ3B1L2RybS90dG0vdHRtX3Bh Z2VfYWxsb2MuYwo+Pj4gaW5kZXggYjQwYTQ2NzhjMjk2Li5lY2QxODMxYTFkMzggMTAwNjQ0Cj4+ PiAtLS0gYS9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV9wYWdlX2FsbG9jLmMKPj4+ICsrKyBiL2Ry aXZlcnMvZ3B1L2RybS90dG0vdHRtX3BhZ2VfYWxsb2MuYwo+Pj4gQEAgLTQyLDcgKzQyLDcgQEAK Pj4+ICAgICNpbmNsdWRlIDxsaW51eC9zZXFfZmlsZS5oPiAvKiBmb3Igc2VxX3ByaW50ZiAqLwo+ Pj4gICAgI2luY2x1ZGUgPGxpbnV4L3NsYWIuaD4KPj4+ICAgICNpbmNsdWRlIDxsaW51eC9kbWEt bWFwcGluZy5oPgo+Pj4gLQo+Pj4gKyNpbmNsdWRlIDxsaW51eC9tZW1jb250cm9sLmg+Cj4+PiAg ICAjaW5jbHVkZSA8bGludXgvYXRvbWljLmg+Cj4+Pgo+Pj4gICAgI2luY2x1ZGUgPGRybS90dG0v dHRtX2JvX2RyaXZlci5oPgo+Pj4gQEAgLTEwNDUsNiArMTA0NSwxMSBAQCB0dG1fcG9vbF91bnBv cHVsYXRlX2hlbHBlcihzdHJ1Y3QgdHRtX3R0ICp0dG0sIHVuc2lnbmVkIG1lbV9jb3VudF91cGRh dGUpCj4+PiAgICAgICAgdHRtX3B1dF9wYWdlcyh0dG0tPnBhZ2VzLCB0dG0tPm51bV9wYWdlcywg dHRtLT5wYWdlX2ZsYWdzLAo+Pj4gICAgICAgICAgICAgICAgICAgICAgdHRtLT5jYWNoaW5nX3N0 YXRlKTsKPj4+ICAgICAgICB0dG0tPnN0YXRlID0gdHRfdW5wb3B1bGF0ZWQ7Cj4+PiArCj4+PiAr I2lmZGVmIENPTkZJR19NRU1DRwo+Pj4gKyAgICAgaWYgKHR0bS0+bWVtY2cpCj4+PiArICAgICAg ICAgICAgIG1lbV9jZ3JvdXBfdW5jaGFyZ2VfZHJ2bWVtKHR0bS0+bWVtY2csIHR0bS0+bnVtX3Bh Z2VzKTsKPj4+ICsjZW5kaWYKPj4+ICAgIH0KPj4+Cj4+PiAgICBpbnQgdHRtX3Bvb2xfcG9wdWxh dGUoc3RydWN0IHR0bV90dCAqdHRtLCBzdHJ1Y3QgdHRtX29wZXJhdGlvbl9jdHggKmN0eCkKPj4+ IEBAIC0xMDU5LDYgKzEwNjQsMTcgQEAgaW50IHR0bV9wb29sX3BvcHVsYXRlKHN0cnVjdCB0dG1f dHQgKnR0bSwgc3RydWN0IHR0bV9vcGVyYXRpb25fY3R4ICpjdHgpCj4+PiAgICAgICAgaWYgKHR0 bV9jaGVja191bmRlcl9sb3dlcmxpbWl0KG1lbV9nbG9iLCB0dG0tPm51bV9wYWdlcywgY3R4KSkK Pj4+ICAgICAgICAgICAgICAgIHJldHVybiAtRU5PTUVNOwo+Pj4KPj4+ICsjaWZkZWYgQ09ORklH X01FTUNHCj4+PiArICAgICBpZiAodHRtLT5tZW1jZykgewo+Pj4gKyAgICAgICAgICAgICBnZnBf dCBnZnBfZmxhZ3MgPSBHRlBfVVNFUjsKPj4+ICsgICAgICAgICAgICAgaWYgKHR0bS0+cGFnZV9m bGFncyAmIFRUTV9QQUdFX0ZMQUdfTk9fUkVUUlkpCj4+PiArICAgICAgICAgICAgICAgICAgICAg Z2ZwX2ZsYWdzIHw9IF9fR0ZQX1JFVFJZX01BWUZBSUw7Cj4+PiArICAgICAgICAgICAgIHJldCA9 IG1lbV9jZ3JvdXBfY2hhcmdlX2Rydm1lbSh0dG0tPm1lbWNnLCBnZnBfZmxhZ3MsIHR0bS0+bnVt X3BhZ2VzKTsKPj4+ICsgICAgICAgICAgICAgaWYgKHJldCkKPj4+ICsgICAgICAgICAgICAgICAg ICAgICByZXR1cm4gcmV0Owo+Pj4gKyAgICAgfQo+Pj4gKyNlbmRpZgo+Pj4gKwo+Pj4gICAgICAg IHJldCA9IHR0bV9nZXRfcGFnZXModHRtLT5wYWdlcywgdHRtLT5udW1fcGFnZXMsIHR0bS0+cGFn ZV9mbGFncywKPj4+ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHR0bS0+Y2FjaGluZ19zdGF0 ZSk7Cj4+PiAgICAgICAgaWYgKHVubGlrZWx5KHJldCAhPSAwKSkgewo+Pj4gZGlmZiAtLWdpdCBh L2RyaXZlcnMvZ3B1L2RybS90dG0vdHRtX3R0LmMgYi9kcml2ZXJzL2dwdS9kcm0vdHRtL3R0bV90 dC5jCj4+PiBpbmRleCBlMGU5YjRmNjlkYjYuLjFhY2IxNTMwODRlMSAxMDA2NDQKPj4+IC0tLSBh L2RyaXZlcnMvZ3B1L2RybS90dG0vdHRtX3R0LmMKPj4+ICsrKyBiL2RyaXZlcnMvZ3B1L2RybS90 dG0vdHRtX3R0LmMKPj4+IEBAIC0yMzMsNiArMjMzLDkgQEAgdm9pZCB0dG1fdHRfaW5pdF9maWVs ZHMoc3RydWN0IHR0bV90dCAqdHRtLCBzdHJ1Y3QgdHRtX2J1ZmZlcl9vYmplY3QgKmJvLAo+Pj4g ICAgICAgIHR0bS0+c3RhdGUgPSB0dF91bnBvcHVsYXRlZDsKPj4+ICAgICAgICB0dG0tPnN3YXBf c3RvcmFnZSA9IE5VTEw7Cj4+PiAgICAgICAgdHRtLT5zZyA9IGJvLT5zZzsKPj4+ICsjaWZkZWYg Q09ORklHX01FTUNHCj4+PiArICAgICB0dG0tPm1lbWNnID0gYm8tPm1lbWNnOwo+Pj4gKyNlbmRp Zgo+Pj4gICAgfQo+Pj4KPj4+ICAgIGludCB0dG1fdHRfaW5pdChzdHJ1Y3QgdHRtX3R0ICp0dG0s IHN0cnVjdCB0dG1fYnVmZmVyX29iamVjdCAqYm8sCj4+PiBkaWZmIC0tZ2l0IGEvaW5jbHVkZS9k cm0vdHRtL3R0bV9ib19hcGkuaCBiL2luY2x1ZGUvZHJtL3R0bS90dG1fYm9fYXBpLmgKPj4+IGlu ZGV4IDY1ZTM5OWQyODBmNy4uOTVhMDhlODFhNzNlIDEwMDY0NAo+Pj4gLS0tIGEvaW5jbHVkZS9k cm0vdHRtL3R0bV9ib19hcGkuaAo+Pj4gKysrIGIvaW5jbHVkZS9kcm0vdHRtL3R0bV9ib19hcGku aAo+Pj4gQEAgLTU0LDYgKzU0LDggQEAgc3RydWN0IHR0bV9wbGFjZTsKPj4+Cj4+PiAgICBzdHJ1 Y3QgdHRtX2xydV9idWxrX21vdmU7Cj4+Pgo+Pj4gK3N0cnVjdCBtZW1fY2dyb3VwOwo+Pj4gKwo+ Pj4gICAgLyoqCj4+PiAgICAgKiBzdHJ1Y3QgdHRtX2J1c19wbGFjZW1lbnQKPj4+ICAgICAqCj4+ PiBAQCAtMTgwLDYgKzE4Miw5IEBAIHN0cnVjdCB0dG1fYnVmZmVyX29iamVjdCB7Cj4+PiAgICAg ICAgdm9pZCAoKmRlc3Ryb3kpIChzdHJ1Y3QgdHRtX2J1ZmZlcl9vYmplY3QgKik7Cj4+PiAgICAg ICAgdW5zaWduZWQgbG9uZyBudW1fcGFnZXM7Cj4+PiAgICAgICAgc2l6ZV90IGFjY19zaXplOwo+ Pj4gKyNpZmRlZiBDT05GSUdfTUVNQ0cKPj4+ICsgICAgIHN0cnVjdCBtZW1fY2dyb3VwICptZW1j ZzsKPj4+ICsjZW5kaWYKPj4+Cj4+PiAgICAgICAgLyoqCj4+PiAgICAgICAgKiBNZW1iZXJzIG5v dCBuZWVkaW5nIHByb3RlY3Rpb24uCj4+PiBkaWZmIC0tZ2l0IGEvaW5jbHVkZS9kcm0vdHRtL3R0 bV90dC5oIGIvaW5jbHVkZS9kcm0vdHRtL3R0bV90dC5oCj4+PiBpbmRleCBjMGU5MjhhYmY1OTIu LjEwZmI1YTU1N2I5NSAxMDA2NDQKPj4+IC0tLSBhL2luY2x1ZGUvZHJtL3R0bS90dG1fdHQuaAo+ Pj4gKysrIGIvaW5jbHVkZS9kcm0vdHRtL3R0bV90dC5oCj4+PiBAQCAtMzMsNiArMzMsNyBAQCBz dHJ1Y3QgdHRtX3R0Owo+Pj4gICAgc3RydWN0IHR0bV9tZW1fcmVnOwo+Pj4gICAgc3RydWN0IHR0 bV9idWZmZXJfb2JqZWN0Owo+Pj4gICAgc3RydWN0IHR0bV9vcGVyYXRpb25fY3R4Owo+Pj4gK3N0 cnVjdCBtZW1fY2dyb3VwOwo+Pj4KPj4+ICAgICNkZWZpbmUgVFRNX1BBR0VfRkxBR19XUklURSAg ICAgICAgICAgKDEgPDwgMykKPj4+ICAgICNkZWZpbmUgVFRNX1BBR0VfRkxBR19TV0FQUEVEICAg ICAgICAgKDEgPDwgNCkKPj4+IEBAIC0xMTYsNiArMTE3LDkgQEAgc3RydWN0IHR0bV90dCB7Cj4+ PiAgICAgICAgICAgICAgICB0dF91bmJvdW5kLAo+Pj4gICAgICAgICAgICAgICAgdHRfdW5wb3B1 bGF0ZWQsCj4+PiAgICAgICAgfSBzdGF0ZTsKPj4+ICsjaWZkZWYgQ09ORklHX01FTUNHCj4+PiAr ICAgICBzdHJ1Y3QgbWVtX2Nncm91cCAqbWVtY2c7Cj4+PiArI2VuZGlmCj4+PiAgICB9Owo+Pj4K Pj4+ICAgIC8qKgo+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwo+PiBkcmktZGV2ZWwgbWFpbGluZyBsaXN0Cj4+IGRyaS1kZXZlbEBsaXN0cy5mcmVlZGVz a3RvcC5vcmcKPj4gaHR0cHM6Ly9uYW0xMS5zYWZlbGlua3MucHJvdGVjdGlvbi5vdXRsb29rLmNv bS8/dXJsPWh0dHBzJTNBJTJGJTJGbGlzdHMuZnJlZWRlc2t0b3Aub3JnJTJGbWFpbG1hbiUyRmxp c3RpbmZvJTJGZHJpLWRldmVsJmFtcDtkYXRhPTAyJTdDMDElN0NjaHJpc3RpYW4ua29lbmlnJTQw YW1kLmNvbSU3QzVkM2I3MGE0M2I4MDQ0NGM1NTA4MDhkNzljODllOTY4JTdDM2RkODk2MWZlNDg4 NGU2MDhlMTFhODJkOTk0ZTE4M2QlN0MwJTdDMCU3QzYzNzE0OTk4ODQ2Njg1Mzc2MiZhbXA7c2Rh dGE9bmkza3U3bkMlMkZENUU4a2l2cHBmdXVGN1pvaXlmTFE4TDNZNGo5SWZIWVVVJTNEJmFtcDty ZXNlcnZlZD0wCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcK aHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?Christian_K=c3=b6nig?= Subject: Re: [PATCH RFC 3/3] drm/ttm: support memcg for ttm_tt Date: Sun, 19 Jan 2020 14:03:58 +0100 Message-ID: <0164fcbf-393a-237f-69d7-aa26dabe6ad9@amd.com> References: <20200113153543.24957-1-qiang.yu@amd.com> <20200113153543.24957-4-qiang.yu@amd.com> Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Lc3KPrSxN4ypxUGuaIhLzwAb/+yrCobaCHQgDGRk9Q=; b=iWTCYkYeY7NTtgfDfHmx4h66voULX2xbgR06XKj/F0tgzopXvax2aWlCfEuYAGwKNR/mKmJzVVFS8QvbjfEjtneF4zDzuP0mzEfrpFusRQZeNd/Yh8faEggDGYTeM0CUnS8VUzasAYQlVASKHlhMq0NiIMUgWYP/XbS1Zx46RKQ= In-Reply-To: Content-Language: en-US Sender: cgroups-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-ID: Content-Type: text/plain; charset="iso-8859-1"; format="flowed" To: Qiang Yu Cc: Qiang Yu , Linux Memory Management List , cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, dri-devel , David Airlie , Kenny Ho , Michal Hocko , Huang Rui , Johannes Weiner , Tejun Heo , Andrew Morton Am 19.01.20 um 03:47 schrieb Qiang Yu: > On Mon, Jan 13, 2020 at 11:56 PM Christian K=C3=B6nig > wrote: >> Am 13.01.20 um 16:35 schrieb Qiang Yu: >>> Charge TTM allocated system memory to memory cgroup which will >>> limit the memory usage of a group of processes. >> NAK to the whole approach. This belongs into the GEM or driver layer, >> but not into TTM. >> > Sorry for responding late. > > GEM layer seems not a proper place to handle this as: > 1. it is not aware of the back storage (system mem or device mem) unless > we add this information up to GEM which I think is not appropriate > 2. system memory allocated by GEM with drm_gem_get_pages() is already > charged to memcg, it's only the ttm system memory not charged to memcg The key point is that we already discussed this on the mailing list and=20 GEM was agreed on to be the right place for this. That's the reason why the Intel developers already proposed a way to=20 expose the buffer location in GEM. Please sync up with Kenny who is leading the development efforts and=20 with the Intel developers before warming up an old discussion again. Adding that to TTM is an absolute no-go from my maintainers perspective. > > Implement in driver like amdgpu is an option. But seems the problem is in= side > TTM which does not charge pages allocated by itself to memcg, won't it be > better to solve it in TTM so that all drivers using it can benefit? Or yo= u just > think we should not rely on memcg for GPU system memory limitation? > >>> The memory is always charged to the control group of task which >>> create this buffer object and when it's created. For example, >>> when a buffer is created by process A and exported to process B, >>> then process B populate this buffer, the memory is still charged >>> to process A's memcg; if a buffer is created by process A when in >>> memcg B, then A is moved to memcg C and populate this buffer, it >>> will charge memcg B. >> This is actually the most common use case for graphics application where >> the X server allocates most of the backing store. >> >> So we need a better handling than just accounting the memory to whoever >> allocated it first. >> > You mean the application based on DRI2 and X11 protocol draw? I think this > is still reasonable to charge xserver for the memory, because xserver all= ocate > the buffer and share to application which is its design and implementation > nature. With DRI3, the buffer is allocated by application, also > suitable for this > approach. That is a way to simplistic. Again we already discussed this and the agreed compromise is to charge=20 the application which is using the memory and not who has allocated it. So you need to add the charge on importing a buffer and not just when it=20 is created. Regards, Christian. > > Regards, > Qiang > >> Regards, >> Christian. >> >>> Signed-off-by: Qiang Yu >>> --- >>> drivers/gpu/drm/ttm/ttm_bo.c | 10 ++++++++++ >>> drivers/gpu/drm/ttm/ttm_page_alloc.c | 18 +++++++++++++++++- >>> drivers/gpu/drm/ttm/ttm_tt.c | 3 +++ >>> include/drm/ttm/ttm_bo_api.h | 5 +++++ >>> include/drm/ttm/ttm_tt.h | 4 ++++ >>> 5 files changed, 39 insertions(+), 1 deletion(-) >>> >>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c >>> index 8d91b0428af1..4e64846ee523 100644 >>> --- a/drivers/gpu/drm/ttm/ttm_bo.c >>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c >>> @@ -42,6 +42,7 @@ >>> #include >>> #include >>> #include >>> +#include >>> >>> static void ttm_bo_global_kobj_release(struct kobject *kobj); >>> >>> @@ -162,6 +163,10 @@ static void ttm_bo_release_list(struct kref *list_= kref) >>> if (!ttm_bo_uses_embedded_gem_object(bo)) >>> dma_resv_fini(&bo->base._resv); >>> mutex_destroy(&bo->wu_mutex); >>> +#ifdef CONFIG_MEMCG >>> + if (bo->memcg) >>> + css_put(&bo->memcg->css); >>> +#endif >>> bo->destroy(bo); >>> ttm_mem_global_free(&ttm_mem_glob, acc_size); >>> } >>> @@ -1330,6 +1335,11 @@ int ttm_bo_init_reserved(struct ttm_bo_device *b= dev, >>> } >>> atomic_inc(&ttm_bo_glob.bo_count); >>> >>> +#ifdef CONFIG_MEMCG >>> + if (bo->type =3D=3D ttm_bo_type_device) >>> + bo->memcg =3D mem_cgroup_driver_get_from_current(); >>> +#endif >>> + >>> /* >>> * For ttm_bo_type_device buffers, allocate >>> * address space from the device. >>> diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc.c b/drivers/gpu/drm/ttm= /ttm_page_alloc.c >>> index b40a4678c296..ecd1831a1d38 100644 >>> --- a/drivers/gpu/drm/ttm/ttm_page_alloc.c >>> +++ b/drivers/gpu/drm/ttm/ttm_page_alloc.c >>> @@ -42,7 +42,7 @@ >>> #include /* for seq_printf */ >>> #include >>> #include >>> - >>> +#include >>> #include >>> >>> #include >>> @@ -1045,6 +1045,11 @@ ttm_pool_unpopulate_helper(struct ttm_tt *ttm, u= nsigned mem_count_update) >>> ttm_put_pages(ttm->pages, ttm->num_pages, ttm->page_flags, >>> ttm->caching_state); >>> ttm->state =3D tt_unpopulated; >>> + >>> +#ifdef CONFIG_MEMCG >>> + if (ttm->memcg) >>> + mem_cgroup_uncharge_drvmem(ttm->memcg, ttm->num_pages); >>> +#endif >>> } >>> >>> int ttm_pool_populate(struct ttm_tt *ttm, struct ttm_operation_ctx *= ctx) >>> @@ -1059,6 +1064,17 @@ int ttm_pool_populate(struct ttm_tt *ttm, struct= ttm_operation_ctx *ctx) >>> if (ttm_check_under_lowerlimit(mem_glob, ttm->num_pages, ctx)) >>> return -ENOMEM; >>> >>> +#ifdef CONFIG_MEMCG >>> + if (ttm->memcg) { >>> + gfp_t gfp_flags =3D GFP_USER; >>> + if (ttm->page_flags & TTM_PAGE_FLAG_NO_RETRY) >>> + gfp_flags |=3D __GFP_RETRY_MAYFAIL; >>> + ret =3D mem_cgroup_charge_drvmem(ttm->memcg, gfp_flags, t= tm->num_pages); >>> + if (ret) >>> + return ret; >>> + } >>> +#endif >>> + >>> ret =3D ttm_get_pages(ttm->pages, ttm->num_pages, ttm->page_flag= s, >>> ttm->caching_state); >>> if (unlikely(ret !=3D 0)) { >>> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c >>> index e0e9b4f69db6..1acb153084e1 100644 >>> --- a/drivers/gpu/drm/ttm/ttm_tt.c >>> +++ b/drivers/gpu/drm/ttm/ttm_tt.c >>> @@ -233,6 +233,9 @@ void ttm_tt_init_fields(struct ttm_tt *ttm, struct = ttm_buffer_object *bo, >>> ttm->state =3D tt_unpopulated; >>> ttm->swap_storage =3D NULL; >>> ttm->sg =3D bo->sg; >>> +#ifdef CONFIG_MEMCG >>> + ttm->memcg =3D bo->memcg; >>> +#endif >>> } >>> >>> int ttm_tt_init(struct ttm_tt *ttm, struct ttm_buffer_object *bo, >>> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h >>> index 65e399d280f7..95a08e81a73e 100644 >>> --- a/include/drm/ttm/ttm_bo_api.h >>> +++ b/include/drm/ttm/ttm_bo_api.h >>> @@ -54,6 +54,8 @@ struct ttm_place; >>> >>> struct ttm_lru_bulk_move; >>> >>> +struct mem_cgroup; >>> + >>> /** >>> * struct ttm_bus_placement >>> * >>> @@ -180,6 +182,9 @@ struct ttm_buffer_object { >>> void (*destroy) (struct ttm_buffer_object *); >>> unsigned long num_pages; >>> size_t acc_size; >>> +#ifdef CONFIG_MEMCG >>> + struct mem_cgroup *memcg; >>> +#endif >>> >>> /** >>> * Members not needing protection. >>> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h >>> index c0e928abf592..10fb5a557b95 100644 >>> --- a/include/drm/ttm/ttm_tt.h >>> +++ b/include/drm/ttm/ttm_tt.h >>> @@ -33,6 +33,7 @@ struct ttm_tt; >>> struct ttm_mem_reg; >>> struct ttm_buffer_object; >>> struct ttm_operation_ctx; >>> +struct mem_cgroup; >>> >>> #define TTM_PAGE_FLAG_WRITE (1 << 3) >>> #define TTM_PAGE_FLAG_SWAPPED (1 << 4) >>> @@ -116,6 +117,9 @@ struct ttm_tt { >>> tt_unbound, >>> tt_unpopulated, >>> } state; >>> +#ifdef CONFIG_MEMCG >>> + struct mem_cgroup *memcg; >>> +#endif >>> }; >>> >>> /** >> _______________________________________________ >> dri-devel mailing list >> dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org >> https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flist= s.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&data=3D02%7C01%7Cchr= istian.koenig%40amd.com%7C5d3b70a43b80444c550808d79c89e968%7C3dd8961fe4884e= 608e11a82d994e183d%7C0%7C0%7C637149988466853762&sdata=3Dni3ku7nC%2FD5E8= kivppfuuF7ZoiyfLQ8L3Y4j9IfHYUU%3D&reserved=3D0