From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 184CEC433E1 for ; Wed, 24 Mar 2021 12:00:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 93BE6619FC for ; Wed, 24 Mar 2021 12:00:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93BE6619FC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 18F716B02A6; Wed, 24 Mar 2021 08:00:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 140326B02A7; Wed, 24 Mar 2021 08:00:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E88176B02A8; Wed, 24 Mar 2021 08:00:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id C63126B02A6 for ; Wed, 24 Mar 2021 08:00:41 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 800AE180ACC3C for ; Wed, 24 Mar 2021 12:00:41 +0000 (UTC) X-FDA: 77954625882.07.250B56D Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2075.outbound.protection.outlook.com [40.107.237.75]) by imf09.hostedemail.com (Postfix) with ESMTP id 91B0D601D8D1 for ; Wed, 24 Mar 2021 12:00:38 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Idet6DHDpgIxWhOGSKt49muu19tTfS9j5t9h0VHPHp4zFQghdQw+fOtqJiyRXnvUQcGu1kprj7wXHyBv4/yWIjutL4w4dAu3QjrfICvD8mrchljUKrcQm76ppiiWlpMr06gfPSv/oJDw33I/NR43OVdMO0D9wASdwWhQgBeylzst3AtsHm9TbfmWFfWGJRA0dFOHJDQmny0OxOA8fFV67EBgRPjRmnzMhfIUGuXYTyG3NelT9GAh/OudQaCF8RriEA8T7UTpT4rTMilahe1yRPqx582Pq1zJwePny6PAwjOP1l1Wbqg4LyoMU/rlNzQfz/goq/KwH0qBUF5e+9oVnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MWNdzbuoxglQZ1ll1ApuUbwwSvP6rBmdY/JLURACrWE=; b=DbG4fMXzNEifY8m8e4ZA3jpOwaxPm7b20+5Ce6SlLnV2R77QQS9Gf4+pBvF8hS4kWXoqwfwp2FZUF8tHQDF6MCAZrP7qQT6PBb40bEn6faIFT2Qmqo8eYuaRKPHcrayDLl8A1ENOnSRcwNNAfWLWgtcjPvcM7jiwzVb0gpqqHvHjROw5z4Elso1NRuQZf2EpaUbuhbTiZzbp62WD/yzwubRLLHa4YlE+gMJ1fD/rJUzyZ0nANWh1MFt9Cx4mAI9y0GMwK4JU2+8x5vwJ1NGhaDkw+9Lf123+neRVqLO0agyEhsg3NgDTS6/V737ouNp+/Ocw2tNMc96YRLF4U6NooA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MWNdzbuoxglQZ1ll1ApuUbwwSvP6rBmdY/JLURACrWE=; b=FA1faFmLCqYeYy/NyXT/P82ftB/IvXnFwiXbOJOR4UXP1mhr4vru0CclckUMRCcZkXFJrp6pETGgAvX5iDplQ86HkZRjgu7xFSVH+9hTCNYDIHH/a2IRMDedvyrl3WtDB/wi23r1gq9PJ/PuiiWYNmxpfqluk8k2DtDq890hpw8= Authentication-Results: amd.com; dkim=none (message not signed) header.d=none;amd.com; dmarc=none action=none header.from=amd.com; Received: from MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) by MN2PR12MB3680.namprd12.prod.outlook.com (2603:10b6:208:169::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Wed, 24 Mar 2021 12:00:35 +0000 Received: from MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::c1ff:dcf1:9536:a1f2]) by MN2PR12MB3775.namprd12.prod.outlook.com ([fe80::c1ff:dcf1:9536:a1f2%2]) with mapi id 15.20.3955.027; Wed, 24 Mar 2021 12:00:35 +0000 Subject: Re: [PATCH] drm/ttm: stop warning on TT shrinker failure To: =?UTF-8?Q?Thomas_Hellstr=c3=b6m_=28Intel=29?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu References: <75ff80c5-a054-d13d-85c1-0040addb45d2@gmail.com> <20808d08-b66c-13c3-f672-ebce216b2fa2@gmail.com> <03889c00-bb5d-ef20-12c6-7e77df073dd9@amd.com> <762c4597-e9bd-6d8d-51b5-16b04f913eb8@shipmail.org> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: <488c8996-1dd2-4928-a98a-4e72f3e0af64@amd.com> Date: Wed, 24 Mar 2021 13:00:28 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Originating-IP: [2a02:908:1252:fb60:4ff1:6377:7c5c:81d9] X-ClientProxiedBy: AM0PR04CA0040.eurprd04.prod.outlook.com (2603:10a6:208:1::17) To MN2PR12MB3775.namprd12.prod.outlook.com (2603:10b6:208:159::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2a02:908:1252:fb60:4ff1:6377:7c5c:81d9] (2a02:908:1252:fb60:4ff1:6377:7c5c:81d9) by AM0PR04CA0040.eurprd04.prod.outlook.com (2603:10a6:208:1::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Wed, 24 Mar 2021 12:00:33 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: f2998f32-ab96-4794-db24-08d8eebc6e8f X-MS-TrafficTypeDiagnostic: MN2PR12MB3680: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BF20NdcS7AFpwBXyOlGXzIFsTUGrZwWDnTpV/KZ9jC6kcOUfTasYWCkqD/D8ayDFLt92s0bTNPOjEztFr/n6JPAlA2FdWGaAcJZJb+tODe4DYOVrsWCx4n+wrPZE0PgdpC0gH5tt8cqCmUxsFVzXUknSuTCFANLkMo46/Ade3aIwEeh3nU4fuz10KjCGA+kZJBuhiGbYINeKDZFGVIHDLAPA2VDy+eTX5tiChM8OJc3JccHkpBpE86COZieP05Ck80/lWtbEF+Vg3KW/c2ru09Rv8OiDEC5RWfoz1TbXgdU8t1vlce/hMZeio2wLjlR95xAoHRzPEz7LJqLiH5u6R9ZgEuO8EBn39op/FvGoGviwDmcwCHSk5gu1eZ+RMD+VU4ij3MThokDi3EC/L5imW9eA1zWGIC2cbQYLlNtA3FJ2tav+IxxdB9LEFzTjg6KkadIs9rrF6gbLdTxQSUXUOexlaoCjsu4R0qzx02DvQqXrvfoGCQHduOs7ojXGEeIzswNN+T/aKEV7KD2c4eA58wVvsGrY0sajeFETy11W5kLfthfEyWZraSxukdUHGZxsd9pTXInimbN77aYz/+94gmLkcJnC6jePXmV8+2bi/OpDwnryoVWVQ99W8/xGFOCl5nw5hHzzaI8JyjnyZ0Qv0YK0UYX+h9L6HEfyOVYmahv2eqHhjEZa8zRc+4zjZcRwJNsVkmWJ+dsRiOHx4aP1UoU+9fnJ4RN6hcC8mAQglkk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR12MB3775.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(396003)(39860400002)(376002)(366004)(136003)(346002)(110136005)(52116002)(86362001)(316002)(16526019)(2616005)(5660300002)(53546011)(186003)(6666004)(36756003)(83380400001)(66946007)(478600001)(6486002)(31696002)(8676002)(38100700001)(31686004)(66476007)(66556008)(6636002)(8936002)(66574015)(2906002)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?bnVWd3F1MWQxd3h0dDBKT3NzOXNHV3o3VjhHYlZYeGdkUGtFM3ZJc0hzSTJm?= =?utf-8?B?QnNGSWpWTitLZGtDMnhkSWYzUjhwVFJ4WVUvcVV4RWZacDg1d0VNTXRjUVc3?= =?utf-8?B?MEdsbkVydzRqc25pQ0JIVVVYOUVEOExRQ0ZvZ3VBWjVIaFZwcDN5UU1PNm9G?= =?utf-8?B?cXNzV2EvcFArM2RBWGNzU2JnakJkcnRjMzQyQ2IzZ0JGVVBKVU4wVUIyeTZK?= =?utf-8?B?Qy82bDFrM3l1WDVRb3V2WENWTzBOb3ZrUEx1blVGM003RWxEVHhEZlBwS21T?= =?utf-8?B?T0UyQ24rL3BERGh1R09qNWYwRzdMKysxZlhPTVU1RjZZdEVuYXBCMW0zMzU1?= =?utf-8?B?bUtYRE92ekV1aitEWTZHME43MmV6UDdhMFB2WFJzRW9LSjFpVFljT1FIUmF5?= =?utf-8?B?Y2tKK0UxSy83K2dEOXowWjNmZGNPN1g1Y2RrWnZNU2pTMWFaQ3kyMzRLRkxt?= =?utf-8?B?eTFJb0FPWjZNelJxSzVyWmJUbm1qWHU1dGJrZkZZV2RHT3JRMXpqdCtHQnRH?= =?utf-8?B?U2hoTXh5dERWK0VJSEN4NEUrUW5pZlVqVitGdjBOdjBmZWoydHdBV1U1cnZr?= =?utf-8?B?M1ZnZjhtS2NsTGVXdXZpY1pXTDFIRytWZWdLcHFpZzNLeStYT0txejdaTXpM?= =?utf-8?B?K05tUklMZDA2dFBkSFJUNlB0ekNQQnREbDFOb1NvMGhTM3lBMFdNWC9XTVBs?= =?utf-8?B?S0ZxU1U4VlFKaXFScnJlVnlydmU4T0NZUlppMGZYWXVidC9XWnhWSkJkSzIy?= =?utf-8?B?ZTFUK1pvOWJteWYvdTVoOTZDT3UraU1YMDB0cDczZEljU0xQQVZtTVhIYWtt?= =?utf-8?B?Q1hHM3JMZmRmdVJmNGo3N0lKMG5NKzU4RWdFTkZ5NWlsUC9sSzh6aW9kSkUy?= =?utf-8?B?VlZBUGVCNHJ5eTFXSitjR1ExZHNBQlYrTTZCRXAzTE96bnplVTY3RFpnOTUz?= =?utf-8?B?UTJjMmxBU0VMUi9pZEhsYk1KYTJJSXRVcVlaak1VdzBDaUdsbjRPRUNubDg2?= =?utf-8?B?c2pZVUNHZFpmNG8xUTZ5Wi85cFBwNVFPZWQ4VkpkQUkxWWRWY3dDTTFpVWxh?= =?utf-8?B?aEJIYWdueDEwUWJHci9CcFVlR1FVRE1kVmEvK25ISVA0eUthQ1FQWWZYTzUz?= =?utf-8?B?MTg5YXFob1lBZ1A1V043WHpEc1FjTWpwWitPRXUvRDlXTGM1aU0wczY2Z0RI?= =?utf-8?B?d1dHN3RDNnBCamxZRjRUM1RFM1VHR2I0dXk5MGRqa2RWTDlWRFlrd2R2SWRL?= =?utf-8?B?bDVBR1ZVME0yM29USHQ2WjBhWk1rd0dmRmJnU2tueE9xVWRkWDV2ajF2aEg5?= =?utf-8?B?bFdpd1RReUpncUNDUHRITzNKaUJLalAvd055M1dlRHo1eHFSQVJPKy9BOFBk?= =?utf-8?B?dk9ha3FPNUtHeEJlLytJS3lyQ0g4bnhTV1YrTklKUEVzaVlVNkt5WGFyYi9U?= =?utf-8?B?ZCswNk82blFEaUVvdW45bVpRMmU5ZW5tazNtU05pa1dwZmxxdExKTWc1bFVB?= =?utf-8?B?RnBjdG4xUHhFMXdTeTN1blNuUUV0UStBdER2aFBTbnZLdTl0TkNESS9tOFc4?= =?utf-8?B?NE9QbDBXOTFwVlU0YjBld1JrRnhFV1hyOGVnelFvWEJ2UDBVUkRvSVdHRTZz?= =?utf-8?B?SW5oM011RDRmcXdmT1hXRFNsUXRwYjVpRUpzUWFsNUpuMVVqNlVLQTM3a3A5?= =?utf-8?B?RFhjRmRrY3V0MjV4WVZlUEduWlpFNkJwVFJlczF5dkFJYVdmaVphUHlvdG44?= =?utf-8?B?MVJBYTMxWTIwYm9lVE10cEtkalV6dXBlSk5LcEpPdnpaQzdJa2tJM1JYSXFl?= =?utf-8?B?OGdxQ1padmhRTHEwS0c0U3dWb0l1YXByZEVTV1RZTi9PUTNKaHo5QkRSTkV0?= =?utf-8?Q?uC3y183lAAwCz?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: f2998f32-ab96-4794-db24-08d8eebc6e8f X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3775.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2021 12:00:35.2187 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: US6JjNfMNUxGoWA35fiYdflTXiTTKiEulUK5lPqVfpzz4yVpxS86Uly7YkeKk87P X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3680 X-Stat-Signature: rzqkchxmf7xzdgapb5enrtc1h1dauksh X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 91B0D601D8D1 Received-SPF: none (amd.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=NAM12-BN8-obe.outbound.protection.outlook.com; client-ip=40.107.237.75 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616587238-812872 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Am 24.03.21 um 12:55 schrieb Daniel Vetter: > On Wed, Mar 24, 2021 at 11:19:13AM +0100, Thomas Hellstr=C3=B6m (Intel)= wrote: >> On 3/23/21 4:45 PM, Christian K=C3=B6nig wrote: >>> Am 23.03.21 um 16:13 schrieb Michal Hocko: >>>> On Tue 23-03-21 14:56:54, Christian K=C3=B6nig wrote: >>>>> Am 23.03.21 um 14:41 schrieb Michal Hocko: >>>> [...] >>>>>> Anyway, I am wondering whether the overall approach is >>>>>> sound. Why don't >>>>>> you simply use shmem as your backing storage from the >>>>>> beginning and pin >>>>>> those pages if they are used by the device? >>>>> Yeah, that is exactly what the Intel guys are doing for their >>>>> integrated >>>>> GPUs :) >>>>> >>>>> Problem is for TTM I need to be able to handle dGPUs and those have= all >>>>> kinds of funny allocation restrictions. In other words I need to >>>>> guarantee >>>>> that the allocated memory is coherent accessible to the GPU >>>>> without using >>>>> SWIOTLB. >>>>> >>>>> The simple case is that the device can only do DMA32, but you also = got >>>>> device which can only do 40bits or 48bits. >>>>> >>>>> On top of that you also got AGP, CMA and stuff like CPU cache behav= ior >>>>> changes (write back vs. write through, vs. uncached). >>>> OK, so the underlying problem seems to be that gfp mask (thus >>>> mapping_gfp_mask) cannot really reflect your requirements, right?=C2= =A0 Would >>>> it help if shmem would allow to provide an allocation callback to >>>> override alloc_page_vma which is used currently? I am pretty sure th= ere >>>> will be more to handle but going through shmem for the whole life ti= me >>>> is just so much easier to reason about than some tricks to abuse shm= em >>>> just for the swapout path. >>> Well it's a start, but the pages can have special CPU cache settings.= So >>> direct IO from/to them usually doesn't work as expected. >>> >>> Additional to that for AGP and CMA I need to make sure that I give th= ose >>> pages back to the relevant subsystems instead of just dropping the pa= ge >>> reference. >>> >>> So I would need to block for the swapio to be completed. >>> >>> Anyway I probably need to revert those patches for now since this isn= 't >>> working as we hoped it would. >>> >>> Thanks for the explanation how stuff works here. >> Another alternative here that I've tried before without being successf= ul >> would perhaps be to drop shmem completely and, if it's a normal page (= no dma >> or funny caching attributes) just use add_to_swap_cache()? If it's som= ething >> else, try alloc a page with relevant gfp attributes, copy and >> add_to_swap_cache()? Or perhaps that doesn't work well from a shrinker >> either? > So before we toss everything and go an a great rewrite-the-world tour, > what if we just try to split up big objects. So for objects which are > bigger than e.g. 10mb > > - move them to a special "under eviction" list > - keep a note how far we evicted thus far > - interleave allocating shmem pages, copying data and releasing the ttm > backing store on a chunk basis (maybe 10mb or whatever, tuning tbh) > > If that's not enough, occasionally break out of the shrinker entirely s= o > other parts of reclaim can reclaim the shmem stuff. But just releasing = our > own pages as we go should help a lot I think. Yeah, the later is exactly what I was currently prototyping. I just didn't used a limit but rather a only partially evicted BOs list=20 which is used when we fail to allocate a page. For the 5.12 cycle I think we should just go back to a hard 50% limit=20 for now and then resurrect this when we have solved the issues. Christian. > -Daniel