From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3AEAC433DB for ; Wed, 24 Mar 2021 12:24:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8198261A06 for ; Wed, 24 Mar 2021 12:24:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8198261A06 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 19BDE6B02BE; Wed, 24 Mar 2021 08:24:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 174CA6B02BF; Wed, 24 Mar 2021 08:24:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F09716B02C0; Wed, 24 Mar 2021 08:24:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0087.hostedemail.com [216.40.44.87]) by kanga.kvack.org (Postfix) with ESMTP id D2C356B02BE for ; Wed, 24 Mar 2021 08:24:36 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 90EECA8E3 for ; Wed, 24 Mar 2021 12:24:36 +0000 (UTC) X-FDA: 77954686152.16.FF3B8F9 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2052.outbound.protection.outlook.com [40.107.92.52]) by imf26.hostedemail.com (Postfix) with ESMTP id B2D22407F8F7 for ; Wed, 24 Mar 2021 12:24:32 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IDgxdvXrd4i729NktA7TLfhfZLbk+gvWvcMrYwNEFpMPX2wE2wYKU+iu9++a8cwvWNhSRs5x71PQNr8g+fik01Gn/gM5uCO8JKaXzW5iRhmN8bJIz5I/FSauL6xN3XBBaktT+KL+WnWK4+H4DOybm/eq1CJAGzYYE9JDL1niCGW42ZNGTcbQlTeLy2dw7jSGlXiwZPYGi5KP0kSAITchKPeoepPYOIoFC1r6bAtGEv9rPQvwdcoL22I0WkUO9luchslOSGwCBQBHYyWlkEKoWPJzCB+cGPnAvMzU0/iExrwH5Ssp9OcsWFzuRK7pKlsoCHPbwwULy3JxNMFgpRgT2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+FBWiIXAhofQbkYblUPIbb5ol27zUlwYxHQMNJEB7HU=; b=gLlBkTuYMz6kMviFxH4dqwcOhQ+WrTH1Ujwr6b0Wz4AKaP2aebX0BIF3vB3nXsiii046RdTr7ngntaZbYfdsx0KDfm39HtIL9AlrcWoUnKgn6tz+1yh8eU9WTEGjY1+fjZRqoZHJ+liaR3C2lTp4//0aq2XGgoSx7VoFGp6ajUczWYXc9/07q1HMwY7BkDRgAuaiYzwReafliqjkECDbgF2vBAPxKCTl0B/3lxhWbRMU5NYlvP1k+RSHjGizMgTJUEJR+5kMjAiWY/2Zjf6QwHNobBhH1dxQfEVZ12eL3HDc1T6oGb2wnIC2XJy1zC6GW2ivz4SAiOTuS6XajwJwMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+FBWiIXAhofQbkYblUPIbb5ol27zUlwYxHQMNJEB7HU=; b=fjpGwJiNXO6YloZaU/yU58DhJXfUtorzNIsWDkjFfP9Tave5iUHNDrWewjhSF4mHkyPNPeSlq23GUhkCUGkfS3Cgb3VtmEvIvFgit86dkcBsvQeAt9hlgwj0lknP2Or+S5z17FcNQBf2/I/MJbCezvJx7kuqG9Cl57LvccbS2S5FtoTGE3t0oZuoiDLyS8M9YNvRUEKgMw+6UTnKnIht/oGxn9tjQhddQ98tu3f6viBnJhSJ5lHXS6a4iis3SEoEgPAhTQiYEvPscZNTV1D3yeHmoYEz1RwFKbra7/XLRdTfpPE2NlYhrLTwOjyUs0FpkzYYG71uv/4rm838yGnfmw== Authentication-Results: shipmail.org; dkim=none (message not signed) header.d=none;shipmail.org; dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) by DM5PR12MB1434.namprd12.prod.outlook.com (2603:10b6:3:77::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Wed, 24 Mar 2021 12:24:32 +0000 Received: from DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::1c62:7fa3:617b:ab87]) by DM6PR12MB3834.namprd12.prod.outlook.com ([fe80::1c62:7fa3:617b:ab87%6]) with mapi id 15.20.3955.027; Wed, 24 Mar 2021 12:24:32 +0000 Date: Wed, 24 Mar 2021 09:24:30 -0300 From: Jason Gunthorpe To: Thomas =?utf-8?B?SGVsbHN0csO2bSAoSW50ZWwp?= , David Airlie , linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, Andrew Morton , Christian Koenig Subject: Re: [RFC PATCH 1/2] mm,drm/ttm: Block fast GUP to TTM huge pages Message-ID: <20210324122430.GW2356281@nvidia.com> References: <20210321184529.59006-1-thomas_os@shipmail.org> <20210321184529.59006-2-thomas_os@shipmail.org> <314fc020-d243-dbf0-acb3-ecfcc9c2443c@shipmail.org> <20210323163715.GJ2356281@nvidia.com> <5824b731-ca6a-92fd-e314-d986b6a7b101@shipmail.org> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Originating-IP: [206.223.160.26] X-ClientProxiedBy: YT1PR01CA0088.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2d::27) To DM6PR12MB3834.namprd12.prod.outlook.com (2603:10b6:5:14a::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (206.223.160.26) by YT1PR01CA0088.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2d::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Wed, 24 Mar 2021 12:24:32 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lP2Yo-001yu8-BY; Wed, 24 Mar 2021 09:24:30 -0300 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 471eed84-f9dc-4738-78b7-08d8eebfc748 X-MS-TrafficTypeDiagnostic: DM5PR12MB1434: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zPLZtBABdo5QzJNJUk7YXugEOmGVOvsnyT652lKBf7KdqNPZbwMikaNfRLj64Xh76vydt2npGpJnSrDr3luuEKON9CvAOJ028O3DYbk0ztb+o7rtkfO6cEXP8r/gZvPJjrBNE1oF2iR4wrb845ktZliHz/OJXeKeJgZH4UMiAxsyny7SoL2KRHyfCIIGSihArdDgBhihJPYfKOs8tRoucsgWwDkJMQ/b7NAFl/yPwbCGo29kgHtu0s3jzRMiqLRKI+xb2xwT1slLTjnpQvbhxA9LdDzwvoiTOO1eCtKscyejj8Ra9LYCIaBAKY2D/yF/Hxw1i8McnqzKmH7WRN6msJuhjG86bCspVCsA2+n9SOaaiZ5R2FTRgpAWQzg+XS4/VNfDRAjDX+aCKKcNtRejM2nIB2jdUuFT8YnaYI90M1kFMfh8mokCQmeVSX8Y+uKfatPH6OV+RSCGOK8mqe49b2MUCks8PB1WuMGKBgiUSUvJXr61E+qNhBwcNz7GXrW+Dap5YKoM0E3w8cn4nxsnhgkzgaO1wCnkX2ODEfd9IDmtSUA2X8J9i8aVNpmw1LvG9s15qVGop4KwUQIe1f43CMJBKv4cRiJ6dvjOIqd70hzzyMrMHAT6uzWU8OsAg6sPmr3SpI4VZD8ad5BxMx1U5wdxVPGTffQ+7XWwc137mhOpZNzseUmmgI+1oZ23fBjTZKINOZMdh/9DAJFV/W6r0w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(396003)(136003)(366004)(376002)(346002)(66556008)(83380400001)(1076003)(5660300002)(9786002)(9746002)(478600001)(26005)(186003)(66574015)(86362001)(316002)(426003)(2616005)(110136005)(66476007)(36756003)(38100700001)(8676002)(8936002)(66946007)(53546011)(33656002)(2906002)(21314003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?dFVsZXNEZG5BWmxSZUJXZzdRNzlvc2Voc0JMSThoREhIby9XRGt6YkpFSm53?= =?utf-8?B?RkZBZnV1N1VYV2t2NXhmUU9HTU1Sa043ZWhBT0VKRHhweFJReThzdnErTjNs?= =?utf-8?B?anFONkpkNmtFbDFxNnpRaUFhV29QV29VcnMxZ2xBWTRWb0kxTVdUaEFxcWpt?= =?utf-8?B?TkY0RnZ1eFRzRU00bHVJbGRaUk1GMG83aGNtUUNBdjRZSlpOeTV0b09sZTlw?= =?utf-8?B?RWlrbFZkeEkvT0MxTk5CaUdiQTlFeXd6cVlxWVY2QXd3VG1ucVA4ZitVZUx5?= =?utf-8?B?bzdoSDlHaVo3VXJZSmY5Z0prVDZWajVVMmp0UmxVZlJzWVZvNGVyeUpaMXE5?= =?utf-8?B?VUQ3Nkx0MElPanBMbys1SHBucU8vTzQ3c25CNDJKSnc5cjVENzYrd1c4a0hv?= =?utf-8?B?ZHVOT0s5VndDY3dibnQ1RlN3RnlkM25rTDRYRmdNUGNmOFZCTnBRTGlzMnNB?= =?utf-8?B?SnpxWDczbmwxck9ZTS9EZUFEci9hVnJOeEk4dW91amlxL3Vlcy9Tc3lKU3BG?= =?utf-8?B?cmMyZGVKSkx4VGkvYUQ5RVA4OXB2Mjg3K1Bwenh2MFlzSWJ0OTk0aUtPb1cz?= =?utf-8?B?N1Y5UXJSV01LcmJkZDIvaDd4NWtodWc5V2xwRjQ1MXVBQTFkYWN2RGJab25t?= =?utf-8?B?Q3ZXOExiTjQwNERnQTRQUFdWdytkbnBjUG9PaTR3V3lDZjFYdDk1b3k0RC9p?= =?utf-8?B?cFV4RXVSc1YwSGljeC9QbDlNcllpN1ZzU2Q3bVRJanpwWGppZnh2djZsN1pH?= =?utf-8?B?ZEZHd1RLWWdHdkZrZWI3UkxHWFRjZXVPNnFJZUI1QUR5bkdTdjQ0TTJkR1ZT?= =?utf-8?B?K0Nuc0xtOTQ1TktPZnQxOUdrQ3ZWMFYzdzdHZzVvRURpcHo1aTJ1REg0TFpu?= =?utf-8?B?SlhKZks5V1p4eU1nbWUzZWdrTnlJcVE5Q1c0TnFjT0FEeGdWSjZqdWZhK1pl?= =?utf-8?B?R2tsMFAwMjBtaVh5WTVuRWx3RmZqSVRGMzV0b1FteUc4NDd6eDRxNi91dHBw?= =?utf-8?B?Tml3OEdPNUNuVHNDWm9jU0tXVFRGcGpGZ0o0cXdpUFBGaUFXdFl4SE9sS1hY?= =?utf-8?B?L1dzeE5CV00xU3BKeTNYUzBpZFlwbnRvcjd5c2NDdUlDbGRPanB4UVdXVytK?= =?utf-8?B?S3BLMzkzdjd2ZmJnNjVNUHg3MW9iSDNCd3R3U1dkalNHZVFHSHJvTHkweGFx?= =?utf-8?B?d0JTdnRTaXVJQzVIZHZEbDQyRmhuaUtWa2h0UDJVNHBreFhKWElQK0QzYjFm?= =?utf-8?B?ditqc05aZnZ5dEtpQ1NCWEo4c0NRUlY5OU81Sm9uSzlCZklML3RPdHJtSE8r?= =?utf-8?B?Z2d4U0grdlZqdi9kWG5wb2l3cXBseWNiM05jWkMxMUpaS1drUlZxcnFXU0ZJ?= =?utf-8?B?V1d4amxoRTFrUTA5WXd5UXZLbkFmTU5Mc1NJV2YwdVNHSVJoc0VYSjJ3NGN5?= =?utf-8?B?WmtKczloNUFGa3RUVVJrYkcxaE54b3ZXMk04SlZ4WFpmaUhzNjFCTVVTZ1RB?= =?utf-8?B?UVM4SnB0ZE82QlJlaGRESmdBRjI0dnJrcnRhZWliZHlGUW9rMklOTTU2WnZO?= =?utf-8?B?ZmVkdkVQWnUrYkVkMWxXRDVuRlZzZm5abk1BNk54RVhTK3Z0SkU1NkRhc0k0?= =?utf-8?B?dVRxSFE2ZEIwR2lQYW1HR09PRmJsTm5pZnB5eVluQmE3VG1YQXF0QlYvMGRP?= =?utf-8?B?TDBUWDltM0Q3c3dENzdPK2IrSXlYTkoxZkV0a3g5Ly83RytZOTJWL0sxK1l2?= =?utf-8?Q?5vpv+hP6EwfF1Y54BmhAl/I4ePCNP6Hpphkhzgj?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 471eed84-f9dc-4738-78b7-08d8eebfc748 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2021 12:24:32.4923 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7/mQ9uFNpgYN4xpXoiphKDjI83MyFxwSWFl8Uz548ioD6d3UWsvH2hzov5XQQmay X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1434 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B2D22407F8F7 X-Stat-Signature: 7kf6h6zcfxas9tszttsbxsyr8z3wfaig Received-SPF: none (nvidia.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=NAM10-BN7-obe.outbound.protection.outlook.com; client-ip=40.107.92.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616588672-869659 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 24, 2021 at 10:56:43AM +0100, Daniel Vetter wrote: > On Tue, Mar 23, 2021 at 06:06:53PM +0100, Thomas Hellstr=C3=B6m (Intel)= wrote: > >=20 > > On 3/23/21 5:37 PM, Jason Gunthorpe wrote: > > > On Tue, Mar 23, 2021 at 05:34:51PM +0100, Thomas Hellstr=C3=B6m (In= tel) wrote: > > >=20 > > > > > > @@ -210,6 +211,20 @@ static vm_fault_t ttm_bo_vm_insert_huge(= struct vm_fault *vmf, > > > > > > if ((pfn & (fault_page_size - 1)) !=3D 0) > > > > > > goto out_fallback; > > > > > > + /* > > > > > > + * Huge entries must be special, that is marking them as de= vmap > > > > > > + * with no backing device map range. If there is a backing > > > > > > + * range, Don't insert a huge entry. > > > > > > + * If this check turns out to be too much of a performance = hit, > > > > > > + * we can instead have drivers indicate whether they may ha= ve > > > > > > + * backing device map ranges and if not, skip this lookup. > > > > > > + */ > > > > > I think we can do this statically: > > > > > - if it's system memory we know there's no devmap for it, and w= e do the > > > > > trick to block gup_fast > > > > Yes, that should work. > > > > > - if it's iomem, we know gup_fast wont work anyway if don't set= PFN_DEV, > > > > > so might as well not do that > > > > I think gup_fast will unfortunately mistake a huge iomem page for= an > > > > ordinary page and try to access a non-existant struct page for it= , unless we > > > > do the devmap trick. > > > >=20 > > > > And the lookup would then be for the rare case where a driver wou= ld have > > > > already registered a dev_pagemap for an iomem area which may also= be mapped > > > > through TTM (like the patch from Felix a couple of weeks ago). If= a driver > > > > can promise not to do that, then we can safely remove the lookup. > > > Isn't the devmap PTE flag arch optional? Does this fall back to not > > > using huge pages on arches that don't support it? > >=20 > > Good point. No, currently it's only conditioned on transhuge page sup= port. > > Need to condition it on also devmap support. > >=20 > > >=20 > > > Also, I feel like this code to install "pte_special" huge pages doe= s > > > not belong in the drm subsystem.. > >=20 > > I could add helpers in huge_memory.c: > >=20 > > vmf_insert_pfn_pmd_prot_special() and > > vmf_insert_pfn_pud_prot_special() >=20 > The somewhat annoying thing is that we'd need an error code so we fall > back to pte fault handling. That's at least my understanding of how > pud/pmd fault handling works. Not sure how awkward that is going to be > with the overall fault handling flow. >=20 > But aside from that I think this makes tons of sense. Why should the driver be so specific? vmf_insert_pfn_range_XXX() And it will figure out the optimal way to build the page tables. Driver should provide the largest physically contiguous range it can Jason