From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,RCVD_ILLEGAL_IP,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D2B1C433E2 for ; Thu, 3 Sep 2020 21:06:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D8052206CA for ; Thu, 3 Sep 2020 21:06:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="aCZBT9i/"; dkim=pass (1024-bit key) header.d=fb.onmicrosoft.com header.i=@fb.onmicrosoft.com header.b="Xv64w5VL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8052206CA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6EDAF6B005D; Thu, 3 Sep 2020 17:06:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 69C936B006C; Thu, 3 Sep 2020 17:06:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 565A76B006E; Thu, 3 Sep 2020 17:06:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 404EA6B005D for ; Thu, 3 Sep 2020 17:06:20 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0601E8245571 for ; Thu, 3 Sep 2020 21:06:20 +0000 (UTC) X-FDA: 77222983320.20.kitty78_5b03d00270ac Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id C26DA180C07A3 for ; Thu, 3 Sep 2020 21:06:19 +0000 (UTC) X-HE-Tag: kitty78_5b03d00270ac X-Filterd-Recvd-Size: 10576 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Sep 2020 21:06:18 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 083L16Hq003689; Thu, 3 Sep 2020 14:06:12 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=date : from : to : cc : subject : message-id : references : content-type : in-reply-to : mime-version; s=facebook; bh=EU35ag4ZOZ8xapaLiJoaGYZ5zsmwEP2lfqFNpkt6XPM=; b=aCZBT9i///WNmodBX9HFp5bBnhgQHdaBNYsbWBV3tmnR134z/rRM/LlnmPZ6C6cmYLuQ j9Y0eRj4H5V+8FxHIUownEujri0arVkUykyyl0uO1zHjT0A71i3tG9BatU1bve1MyknN EzdlZVnCx5BF1Fr611zA+vtYbpGYrplry70= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 33a4cnk0e9-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 03 Sep 2020 14:06:12 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (100.104.31.183) by o365-in.thefacebook.com (100.104.36.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 3 Sep 2020 14:06:08 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RGv36NSbywxDUiscjDK2Cw4Nz8PmQZVfulETW+6zqbGzwQjrM136MRDF7YT19iTVRcEtWLheNSYqUbaCTMWqSSZdvclFH9ST1ibO7/NmQquyGzRTp2lukDiou2gKQpAFV/c59psDK10UKoenyEDSdQDExlV0s0wIiqGPowvbaRfzaswTd61mskCzXOfEaSQDaLg5mZTs6kyTOcVGKmVJ2cXfaB2bq8HApUDAb5gl+rWNcHg7mzVTKLSQ0+W/LUv6mGBsoKyYWQIlN6jSKkyJ1q8woJ2Y3xZwlN0p/YyriBl9MSOT3Sldxfxdx/FfMwoF5msZlzXWwboDc/T+Zn2LIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EU35ag4ZOZ8xapaLiJoaGYZ5zsmwEP2lfqFNpkt6XPM=; b=l2gkpmkHJf30yhBBKexV/GRwk3y2Tv1Ouz4n0NDSwcKckTC6dtwJHt+9XSJMYpE+2y8+Fgv/Vyq51Tj/SQ6SLxYIXaTDSus71iGu4c//SNbXQInUJCRRqrppxD1cuZA85PqpzN301GD5woF/rLLxA3t4jXxnsTUsWV8yyngLOset5xnb3XGFhhyQpmXmaxU/1dqcMF1+HW9Nw+nHQUf060xcLV9aVaKe9vYEpAIyIVz2gUbZd29R1lYZEgbc+B185dFyzAXxM9FddkmWdTG4tc4L1vmetEhCP1IXsK6SVUsD4q8LRD6TZFciC33ePj1flxlQXdEapIUut0eTsHlZ5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fb.com; dmarc=pass action=none header.from=fb.com; dkim=pass header.d=fb.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector2-fb-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EU35ag4ZOZ8xapaLiJoaGYZ5zsmwEP2lfqFNpkt6XPM=; b=Xv64w5VLxkl63ArmmaMxVQZfVD1nYyHxh3kYTJhE3Vjm1gkHek5/7TJkFY3Kezs7mGWf0guVs70Q8MAjG+IgWAwVhFrtXgVv9QA2xzvN9KnEP3RQlQ8rtUrTNd+qbG0hJmE2Xi8d0oSv0Xks7sWRSJfArWMlZG09sdqvk67o27E= Authentication-Results: oracle.com; dkim=none (message not signed) header.d=none;oracle.com; dmarc=none action=none header.from=fb.com; Received: from BYAPR15MB4136.namprd15.prod.outlook.com (2603:10b6:a03:96::24) by BYAPR15MB2869.namprd15.prod.outlook.com (2603:10b6:a03:b3::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15; Thu, 3 Sep 2020 21:06:05 +0000 Received: from BYAPR15MB4136.namprd15.prod.outlook.com ([fe80::354d:5296:6a28:f55e]) by BYAPR15MB4136.namprd15.prod.outlook.com ([fe80::354d:5296:6a28:f55e%6]) with mapi id 15.20.3348.016; Thu, 3 Sep 2020 21:06:05 +0000 Date: Thu, 3 Sep 2020 14:06:01 -0700 From: Roman Gushchin To: Mike Kravetz CC: Michal Hocko , Zi Yan , , Rik van Riel , Kirill A.Shutemov , Matthew Wilcox , Shakeel Butt , Yang Shi , David Nellans , Subject: Re: [RFC PATCH 00/16] 1GB THP support on x86_64 Message-ID: <20200903210601.GI60440@carbon.dhcp.thefacebook.com> References: <20200902180628.4052244-1-zi.yan@sent.com> <20200903073254.GP4617@dhcp22.suse.cz> <20200903162527.GF60440@carbon.dhcp.thefacebook.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: BY3PR05CA0016.namprd05.prod.outlook.com (2603:10b6:a03:254::21) To BYAPR15MB4136.namprd15.prod.outlook.com (2603:10b6:a03:96::24) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from 255.255.255.255 (255.255.255.255) by BY3PR05CA0016.namprd05.prod.outlook.com (2603:10b6:a03:254::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.7 via Frontend Transport; Thu, 3 Sep 2020 21:06:04 +0000 X-Originating-IP: [2620:10d:c090:400::5:a39b] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4aca5c70-9e09-4d9b-bd7f-08d8504d2bc4 X-MS-TrafficTypeDiagnostic: BYAPR15MB2869: X-Microsoft-Antispam-PRVS: X-FB-Source: Internal X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7ItrMZsgCPKCEFwOAaoQ6vK+141gcMd2wTxzYa02me00E49COWF34H15yVbJ7aKr/QJDssKpvsfYx22YOxJcgUs029mCcguT/0sYD1r3YRO+NOgFa5SrAGnMEu/GECCTJJ2Uu9u5i9xWn2N96KOdJ4FJHiVfqzQ0lMKoIYAol2K0p7592hMG0ykUzo7Uc+H0pe1YSSWXvtHNRcXzzm+PjzWuIOYqbN6dy2PlBooDzf4q9Ob/jhOu4N/v62vntGbN3UEBkxCB8JpqYk6f6ArwoaOrbIeeQBXfCevQCm9gAf60uklByC8Idqbi0oBmYBzD8MqvqThg8PSMBGH0iOgN/g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR15MB4136.namprd15.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(396003)(376002)(346002)(39860400002)(4326008)(186003)(2906002)(1076003)(7416002)(8936002)(66476007)(8676002)(66556008)(5660300002)(6916009)(53546011)(66946007)(9686003)(6666004)(478600001)(6486002)(83380400001)(316002)(54906003)(16576012)(956004)(86362001)(52116002)(33656002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData: Vqk81eK7spy59XtGXmCy6mkaDuQxAf5e0hPHmhe/lmrvCJEVF9ePOD3ge4qLzp+xJu0IZf5VcGZkhV46ugQ+XjXo9x+go9iDOCMPsSV5cdsCM9849wKGBotW5ElRGVCgaVCBXagQzQbO7N7GjST162RN6yDaApRuWKeyVZFzDYBsTeWbkMU4fmNPY0kxeLUgspuKoAcihQDw+H5Yn+1OefRDXIuM3yCh7PLP0fpfLW0E/VyDau6LvVkK3naCUP1PKH9XOrBYX9JqInW2iwGLw2PRyrVfqtRXwgd5IEVhL6hwgxE/G1MAFs28XzM6E7qaWlzsTAfHLQtcifrI7DjvJc5gHVegpLz0D/9qyGaC+O1DYcJmwvEWonbv3eC6sHi6TXRoqUWnoFFr17noYG3KQReuej4cLeFtbpa/+r51Mq1RgaZwKhIVzVdIs1lWRZ/5AptL2kEjuGYZXV0b/1+frnJ9aZmkVZ0tPRyPFvQQh2qTC8nt6EgXJN56Cm5OkpegxF3hY/5WbEFiW+4r67CdQi0XNYda9i+Tp5Im/CeJzVMmRPqnR9X7YY94uSRbtQYuLotpvP/rkfvpQIUhY3Zm4cN4ji5VfOa+dxanbuvAH+X8N6JagXAaITzwUYjyf27PLUtNDwZulebEOSp9CaPlXxAQPOR89+DOwEiFEVwgwqg= X-MS-Exchange-CrossTenant-Network-Message-Id: 4aca5c70-9e09-4d9b-bd7f-08d8504d2bc4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Sep 2020 21:06:05.1891 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: z9eKnPgGNdxZQ1FGDrgiCC8s97M9uCqdBEFCVL04ECxRxIeKHqTU9OFGSyHkHPeF X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB2869 X-OriginatorOrg: fb.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-09-03_14:2020-09-03,2020-09-03 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 phishscore=0 malwarescore=0 bulkscore=0 impostorscore=0 priorityscore=1501 mlxlogscore=999 adultscore=0 suspectscore=1 lowpriorityscore=0 spamscore=0 clxscore=1015 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009030188 X-FB-Internal: deliver X-Rspamd-Queue-Id: C26DA180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Sep 03, 2020 at 01:57:54PM -0700, Mike Kravetz wrote: > On 9/3/20 9:25 AM, Roman Gushchin wrote: > > On Thu, Sep 03, 2020 at 09:32:54AM +0200, Michal Hocko wrote: > >> On Wed 02-09-20 14:06:12, Zi Yan wrote: > >>> From: Zi Yan > >>> > >>> Hi all, > >>> > >>> This patchset adds support for 1GB THP on x86_64. It is on top of > >>> v5.9-rc2-mmots-2020-08-25-21-13. > >>> > >>> 1GB THP is more flexible for reducing translation overhead and increasing the > >>> performance of applications with large memory footprint without application > >>> changes compared to hugetlb. > >> > >> Please be more specific about usecases. This better have some strong > >> ones because THP code is complex enough already to add on top solely > >> based on a generic TLB pressure easing. > > > > Hello, Michal! > > > > We at Facebook are using 1 GB hugetlbfs pages and are getting noticeable > > performance wins on some workloads. > > > > Historically we allocated gigantic pages at the boot time, but recently moved > > to cma-based dynamic approach. Still, hugetlbfs interface requires more management > > than we would like to do. 1 GB THP seems to be a better alternative. So I definitely > > see it as a very useful feature. > > > > Given the cost of an allocation, I'm slightly skeptical about an automatic > > heuristics-based approach, but if an application can explicitly mark target areas > > with madvise(), I don't see why it wouldn't work. > > > > In our case we'd like to have a reliable way to get 1 GB THPs at some point > > (usually at the start of an application), and transparently destroy them on > > the application exit. > > Hi Roman, > > In your current use case at Facebook, are you adding 1G hugetlb pages to > the hugetlb pool and then using them within applications? Or, are you > dynamically allocating them at fault time (hugetlb overcommit/surplus)? > > Latency time for use of such pages includes: > - Putting together 1G contiguous > - Clearing 1G memory > > In the 'allocation at fault time' mode you incur both costs at fault time. > If using pages from the pool, your only cost at fault time is clearing the > page. Hi Mike, We're using a pool. Under dynamic I mean that gigantic pages are not allocated at a boot time. Thanks!