From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 463CAC433DB for ; Wed, 24 Mar 2021 12:07:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B6F9E619FE for ; Wed, 24 Mar 2021 12:07:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B6F9E619FE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 334DF6B02B1; Wed, 24 Mar 2021 08:07:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E41C6B02B5; Wed, 24 Mar 2021 08:07:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 110806B02B7; Wed, 24 Mar 2021 08:07:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id E5AD66B02B1 for ; Wed, 24 Mar 2021 08:07:52 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 89A0D181AEF3F for ; Wed, 24 Mar 2021 12:07:52 +0000 (UTC) X-FDA: 77954643984.31.6170415 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2069.outbound.protection.outlook.com [40.107.223.69]) by imf28.hostedemail.com (Postfix) with ESMTP id 18076200024F for ; Wed, 24 Mar 2021 12:07:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QtGkQ7mumxqL2v4MT4MXFv/7Sriv7YMnZ4Kwmj2dxrJ/UmAndXcXjvEjrrY9SFuVk8baXKjVAOMKyNG0jotya5yRbIvKHuXTAY0RO16j6CB2t6nrnxV6VqIPrvMscASiy1ihCxAievwGgn+LJJeiXfnxkg8Ci5mhWmnQPvygLIT5Qr1LcLJ9v94RJ0i0N7yMb9xDvTh3kiZf5EGiTMjenLmFYhXBgMdjiQZStRS6x5PucOkANywB3DgiRZ91XJFMmNKpl8wswYsVw/U4pK22kHEAYJNQKjzjL/CNNgPnW0vvI77tEJIUy8+pQgj5fb49QROlb1ScdslaD06d3vGonw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cvFgveJ2Po7U6S4+J/zubrPhN4AnXEh5YFH36me8VN8=; b=LNPjst1h9xfcpA/hm535oD3nbqPH5ehZFmeR6y0TgpwvLXQJdpDi0p3FKy4KetDfeYMhCdRn3kp12e9wOs7y46YbFQvp43RBB+LZ1WKFav9B6Vy6AqnRn6VHpPUhuoQoteHr1k/KbjRT+/8/UsczOEj5wrtonCSsTrbvf37ZPWVXfXym4No2rdSZaOAivSrf6SD3G7+eVkTKEWPdnf+r8m7uAONS/CYwpwsiJCh5HGIn1qJta0hBEr6WRIRjII0XB+jKi8b5ci+2D/bmHYZUTu8/onM8IXBDq/sCn/zNuR/d4GUGf2cgr1gFpyBTAsX97948DLERk87tD+BHsivu0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cvFgveJ2Po7U6S4+J/zubrPhN4AnXEh5YFH36me8VN8=; b=x2lHFdTamlznouzv/EOWphGZhE1iUjA93BNzQM90GVZ4sX+17mHi13dSrVbxerJR30mFcanuMxgT5b7t4B6covokVyA38K8DCX7LN6tRLjy+iMaIIPfhvDjpE3AaNs0tY2BF6aTvTY+WMb4ir0rTHeelqLdSZvaDU05Zpl1Lj3Q= Authentication-Results: amd.com; dkim=none (message not signed) header.d=none;amd.com; dmarc=none action=none header.from=amd.com; Received: from MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) by MN2PR12MB3885.namprd12.prod.outlook.com (2603:10b6:208:16c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Wed, 24 Mar 2021 12:07:49 +0000 Received: from MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::c1ff:dcf1:9536:a1f2]) by MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::c1ff:dcf1:9536:a1f2%2]) with mapi id 15.20.3955.027; Wed, 24 Mar 2021 12:07:48 +0000 Subject: Re: [PATCH] drm/ttm: stop warning on TT shrinker failure To: =?UTF-8?Q?Thomas_Hellstr=c3=b6m_=28Intel=29?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu References: <75ff80c5-a054-d13d-85c1-0040addb45d2@gmail.com> <20808d08-b66c-13c3-f672-ebce216b2fa2@gmail.com> <03889c00-bb5d-ef20-12c6-7e77df073dd9@amd.com> <762c4597-e9bd-6d8d-51b5-16b04f913eb8@shipmail.org> <488c8996-1dd2-4928-a98a-4e72f3e0af64@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: <31a52f86-e4af-f1d3-90b2-6eff8ec5f300@amd.com> Date: Wed, 24 Mar 2021 13:07:44 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Originating-IP: [2a02:908:1252:fb60:4ff1:6377:7c5c:81d9] X-ClientProxiedBy: AM4PR05CA0018.eurprd05.prod.outlook.com (2603:10a6:205::31) To MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2a02:908:1252:fb60:4ff1:6377:7c5c:81d9] (2a02:908:1252:fb60:4ff1:6377:7c5c:81d9) by AM4PR05CA0018.eurprd05.prod.outlook.com (2603:10a6:205::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25 via Frontend Transport; Wed, 24 Mar 2021 12:07:47 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: cd680673-e4b4-4f32-014f-08d8eebd710a X-MS-TrafficTypeDiagnostic: MN2PR12MB3885: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hpWaS9TkIHO6tYe46fXEpbYofoMYSkDClfLmJAYWM2UHJuBLaI89olJiOjHtb0I5qULrgKYUz40BfFJiRzLNRKKUIN0jz/zXhcCsw4fYlgqME76gKMpoTwROIvar0Z09BRWzKG8LTs3IM8CaXS+qa40xyzVficZGGujvi0RqFPvC820XEZLqPblveyRD6aJDREEPjpbKcNT3+2l4Wy7qQf5WcyuCswxlaDSv1kHJf7xnkRS6V282Ov0b7nWIRYcAYuiPZCyiPId5yfpZ45s9MZRCF2A2zaICnZ2pDp2uI/Dn96/rea0FP1lsY02vY3a9tNFK7ZAASCGXzVAeVufePL9fnbqNAmYP0q9z9IWeeQ66rS5pj2QRYzpC8bTwFDpv+GhRQbsVLQx/8hCw5pcAGwEOwdykASWcCRHGBU5sz59/k4kuJOEB/MjdR3tv+mZcH9leWWMwY0w3wJiPDHqqUa8GUaneg10y70kBAqZRjqcxKDJwabOj3hSR/qvER3X7dgGDT2/ImEs2CfUzCkHxy88pGB9oYh8+uxpnBEljCkMoU7/UtOuVg7kb3WXpkS8rAwe8jBJm7iDPwXK7AUynOKp3HegWaDm/75xW3awd5e/b68lCOG0BhsbhlZSZkRd764ywRLPrs2Q/513F9BPaQBEb3FQOM4IyvUpmUxyH/35F1hP3KZ1Lne8ei8tIR4XTjLUPRYd3rjtDSyp/mNs1fXy1BHlKjgorGz8KDcWpEOo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR12MB3775.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(376002)(346002)(136003)(366004)(396003)(39860400002)(31686004)(478600001)(36756003)(6666004)(8676002)(16526019)(110136005)(5660300002)(316002)(6636002)(53546011)(186003)(6486002)(86362001)(83380400001)(38100700001)(66476007)(52116002)(8936002)(31696002)(66574015)(66946007)(2906002)(2616005)(66556008)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?ckEwUVhMdHRuQ3Y4YmtvWEZRbzBLUmlwOFNtSHFhb0U2NW9EM2lJdFNxUkJu?= =?utf-8?B?bEFsV1lEQ2xLTDhaWjlJK3NGazBaSFJ6Q1h4WnlaUU5wZnZyM25lYmxnZVM4?= =?utf-8?B?K0ZEbkNFOVhzc2wyTnBTdlBhWnY0Q2lWM0JHL3FRL3hTaXpEWjYxZ0JZeE9o?= =?utf-8?B?dTdQS2h3V2lmM1Y4OGx1V0szcE95MklsMkd4MUpjZWNPcUxCOXRrb0RYcVhP?= =?utf-8?B?bUp4SGxaU25VSjBzMHJ3amExelJOWDVncGppT2xvMXdxSlVXNkYybEY2amhL?= =?utf-8?B?TG5xTDJrUTJvaWdwQTlCZ3ZYVzlyMW1IYmpoSVMxRm9Nb2YwSk5pL28vWW5B?= =?utf-8?B?WUxHd0dGK1QvMWIvUFFEYmtvRDRCOS94SmlIelB0T3h2TzZBYVUrdkZGUVVN?= =?utf-8?B?WU9Hb0VONmFFdSsrcWJ4cWs2WlcvQTFkRk82alUrMUlxM1NobkxVTm8rSjcz?= =?utf-8?B?R2w1TFJmclVQUjA1ZXhUTWlKK0xyeGl4bGJYTXFqMjNpSEo2U016Z0RhRTV3?= =?utf-8?B?Y3MzUitHbDBwN3ZPSG9XTmxmSkVJWUVuZGd1QjF0SXc5ZkJES1NkMGd1enBM?= =?utf-8?B?N1ZEcFg4anV0ZU1IMmJPakx4K1F2L2RCTmh6Z3BNY2FaT3NjMyttZFNia1Bo?= =?utf-8?B?c1Q2SHRtSTRuNWE5YjRYSXplSDVPSkV4ejdqclVTcnhBTjFncG1SOUVuZVcy?= =?utf-8?B?WWF2NmhJd2VhRGNOWTBEMXh4NG1VbC95SnhZQThreEkwM1Q1VmZxYzBHYlhi?= =?utf-8?B?YllYYjl1Wis5eGdJeUxBZWZod0lObDN1L0h4L29nQTdZaUhLdnhtQkJXR3Bt?= =?utf-8?B?YjNCT213NVNGS2pQeUc3RVZLSXdHTUgrc04xTkpheHRHUkQ5YnFTcXdiUkdk?= =?utf-8?B?ZUNBTUF5d2RxcU14YWxGSU9BYnBzTlpSZ0hsM2tYaVYrMkpZNUdNRUkwK0ZS?= =?utf-8?B?SnM5MmRmdGNLZVkxMXpSL3hQdzBzYjdVUzkvaWlXSWRQbWN0cnZNR1QwWjYy?= =?utf-8?B?Nkk0YXNFbGZkVTcyT1paSEkrcE5DYkltazZuQTIwZVBsbEdQcEVNK2Y1Yktp?= =?utf-8?B?UXkzY1d2elFocHRXNlVBWnNmWWVUaWJmZzdYbzBRSzZRMGhMLzdBRnRVM3Qr?= =?utf-8?B?dGtaRXVubjVqRXUyR0NuMlRlc0w4NEFodlYvbVp4b1pDN0ZucGtaOEZHSDFk?= =?utf-8?B?K2FZaUNNc2h5MURyZUFKZWJrUFpBNStKdldmWXJ1V0Q5aHhEaW9pQkczcjJt?= =?utf-8?B?ZnlTc2lIbEM1em5YcmtVWGNSR1dhSWJBZUxTcE4za21wOWJvNVJycXlKTWl2?= =?utf-8?B?Ymw4cjdGNkpGQnllSTNEcXUzamx5QmQ2UHFVa0lUSkJRQUtscVNSTFpPbFk4?= =?utf-8?B?RHpOSnhDZ2J4WjVWQlZZNExqYlp0d0xZdEFlS2duNnduLzRRa1JSMUZzSTNU?= =?utf-8?B?MnFRZTk0YUg3dVN3aVdIQ0tBc3NKZjRkckZ2eElEbzFQdHFMMllVVFh2dzhV?= =?utf-8?B?alZJWDlaQlFtQzVGMzJLbkJnWHY3WDdOdjFid0JZd0xWR1J2bGQ5MDQ5Q2E5?= =?utf-8?B?RllmRlJzVmg2dTFidjFNTzg1OFpTWUxEQ2t5cWh4bVlzNS9aSDBnVFNpTTZ4?= =?utf-8?B?aWtkNStPSUlmWldFYXBiQ1hWZGR5ZndkbjhXcUlXS1NmTVVvVUhXUnBtUlpQ?= =?utf-8?B?VnE3Q1BhT0p2NmZPWWpjQ2UxTlUvYXI1TFdMeklLUWwxVlVIMCtkN0xOZnk1?= =?utf-8?B?Q2dNVDFhZFJoczZHTWtVb0gwTDRGTlRNejF3SmNmVDFoUDBhSEc5d3BwWU1a?= =?utf-8?B?Vmk2WlBxbS9DY1hhSVE1NkVxVWhDSDJXRTM3MUIxczFkMUJYTVZUelFVdkZX?= =?utf-8?Q?Bxu/5C0Wtp2vc?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: cd680673-e4b4-4f32-014f-08d8eebd710a X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3775.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2021 12:07:48.8824 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WgiojhdAhIb9Vis0S3BJNx+GrG279dm+oLy3W7XzVyyp0fXQKdq1FXqx9TNE7kN7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3885 X-Stat-Signature: yth95rssf9yj85mztpzp8qubg5gt6fcq X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 18076200024F Received-SPF: none (amd.com>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=NAM11-DM6-obe.outbound.protection.outlook.com; client-ip=40.107.223.69 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616587670-44258 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Am 24.03.21 um 13:01 schrieb Daniel Vetter: > On Wed, Mar 24, 2021 at 01:00:28PM +0100, Christian K=C3=B6nig wrote: >> Am 24.03.21 um 12:55 schrieb Daniel Vetter: >>> On Wed, Mar 24, 2021 at 11:19:13AM +0100, Thomas Hellstr=C3=B6m (Inte= l) wrote: >>>> On 3/23/21 4:45 PM, Christian K=C3=B6nig wrote: >>>>> Am 23.03.21 um 16:13 schrieb Michal Hocko: >>>>>> On Tue 23-03-21 14:56:54, Christian K=C3=B6nig wrote: >>>>>>> Am 23.03.21 um 14:41 schrieb Michal Hocko: >>>>>> [...] >>>>>>>> Anyway, I am wondering whether the overall approach is >>>>>>>> sound. Why don't >>>>>>>> you simply use shmem as your backing storage from the >>>>>>>> beginning and pin >>>>>>>> those pages if they are used by the device? >>>>>>> Yeah, that is exactly what the Intel guys are doing for their >>>>>>> integrated >>>>>>> GPUs :) >>>>>>> >>>>>>> Problem is for TTM I need to be able to handle dGPUs and those ha= ve all >>>>>>> kinds of funny allocation restrictions. In other words I need to >>>>>>> guarantee >>>>>>> that the allocated memory is coherent accessible to the GPU >>>>>>> without using >>>>>>> SWIOTLB. >>>>>>> >>>>>>> The simple case is that the device can only do DMA32, but you als= o got >>>>>>> device which can only do 40bits or 48bits. >>>>>>> >>>>>>> On top of that you also got AGP, CMA and stuff like CPU cache beh= avior >>>>>>> changes (write back vs. write through, vs. uncached). >>>>>> OK, so the underlying problem seems to be that gfp mask (thus >>>>>> mapping_gfp_mask) cannot really reflect your requirements, right?=C2= =A0 Would >>>>>> it help if shmem would allow to provide an allocation callback to >>>>>> override alloc_page_vma which is used currently? I am pretty sure = there >>>>>> will be more to handle but going through shmem for the whole life = time >>>>>> is just so much easier to reason about than some tricks to abuse s= hmem >>>>>> just for the swapout path. >>>>> Well it's a start, but the pages can have special CPU cache setting= s. So >>>>> direct IO from/to them usually doesn't work as expected. >>>>> >>>>> Additional to that for AGP and CMA I need to make sure that I give = those >>>>> pages back to the relevant subsystems instead of just dropping the = page >>>>> reference. >>>>> >>>>> So I would need to block for the swapio to be completed. >>>>> >>>>> Anyway I probably need to revert those patches for now since this i= sn't >>>>> working as we hoped it would. >>>>> >>>>> Thanks for the explanation how stuff works here. >>>> Another alternative here that I've tried before without being succes= sful >>>> would perhaps be to drop shmem completely and, if it's a normal page= (no dma >>>> or funny caching attributes) just use add_to_swap_cache()? If it's s= omething >>>> else, try alloc a page with relevant gfp attributes, copy and >>>> add_to_swap_cache()? Or perhaps that doesn't work well from a shrink= er >>>> either? >>> So before we toss everything and go an a great rewrite-the-world tour= , >>> what if we just try to split up big objects. So for objects which are >>> bigger than e.g. 10mb >>> >>> - move them to a special "under eviction" list >>> - keep a note how far we evicted thus far >>> - interleave allocating shmem pages, copying data and releasing the t= tm >>> backing store on a chunk basis (maybe 10mb or whatever, tuning tb= h) >>> >>> If that's not enough, occasionally break out of the shrinker entirely= so >>> other parts of reclaim can reclaim the shmem stuff. But just releasin= g our >>> own pages as we go should help a lot I think. >> Yeah, the later is exactly what I was currently prototyping. >> >> I just didn't used a limit but rather a only partially evicted BOs lis= t >> which is used when we fail to allocate a page. >> >> For the 5.12 cycle I think we should just go back to a hard 50% limit = for >> now and then resurrect this when we have solved the issues. > Can we do the 50% limit without tossing out all the code we've done thu= s > far? Just so this doesn't get too disruptive. Yeah, I just need to get back to v1 of this patch. Before you convinced=20 me that the shrinker is the better approach .) Cheers, Christian. > -Daniel