From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1D97C433DB for ; Wed, 31 Mar 2021 02:04:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 010C4619BB for ; Wed, 31 Mar 2021 02:04:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 010C4619BB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 511D66B007E; Tue, 30 Mar 2021 22:04:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C2AF6B0081; Tue, 30 Mar 2021 22:04:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29ED86B0082; Tue, 30 Mar 2021 22:04:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 08DE06B007E for ; Tue, 30 Mar 2021 22:04:26 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B922B8249980 for ; Wed, 31 Mar 2021 02:04:25 +0000 (UTC) X-FDA: 77978524890.36.FA4D801 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2066.outbound.protection.outlook.com [40.107.236.66]) by imf17.hostedemail.com (Postfix) with ESMTP id E1D7940002CF for ; Wed, 31 Mar 2021 02:04:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dnahb5ZE34B/k/QTHqcWk9jG3qG8j7MBPwD0PiRr4x25mt1atnT8MMnuz6sWXke2XIWWtft7e1Cl4nKZbR92Z0KtkaSHEec25gLVE/AxG5UT7d9WjAL6B+A3kfsYzulPY+NQXzYRjJwAPc6xKlcF39oXdybCOv1y7YAuWoA3/vN7jBGU0lswE5C56OW0bwlEgvveswIRNMthZhiWJ1AiRvin1ly6RnvTCqR9pDeDVAnb7tA/NrntjU3f02IrQjQsHt1eS4J+Xahm0AsfIfmSbrKzalbD0LdDjvvvZ8DKuTwRVj7Jv6BbNKt1bgnWa/cCDLsluo1QJ8ASBPBm+GP80Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PQJ/g/3vGhGDArGFK06DRXSESN7YVxgtSATG4nx5Ex8=; b=UiRjJP+3CcS6sDVH3+BORCksTuPfZUfgVAiQ1Fnlj4R6qI9gACrF6HqcsYwbDVU06tiGDvrpyr/vyYbcZwkwRRd02HxioMLys4DKTJIFh51s4UnGDM2r3JTTjij5BmX6kXD7geHDFHnM5P8Cl0RCpK3jkRB+Kf0mpodD4xsS7G2yW8NWb+tyF3Ccz80oXekCUEuywcdzCtErCQMfmCLegDkMt24q53VjfysS7WkkNc/OZbxgZ7XUwuf6zASxCODRmMujOmay50Htknf9a+cVc2DAcJe0ZAbFpp9i64p+rnXOF9wDcPie7zklxqKLdjPW7CzM+Ba9bn200i7gNYvJJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PQJ/g/3vGhGDArGFK06DRXSESN7YVxgtSATG4nx5Ex8=; b=rI1b0A9hRgFC2y+CYaCaiaK+sXJr2SzTloWIqh4FMr+lN502gozwMJnR41YKC2B8tmNV3hz41Ne5Ch/4henaTkiCv6aSvPcrmNXJb1aTT0tAFZ8m9ErtfKPefuWm777phGa5uJUFkniAJxt6uzHkH+etuoZ0DHk7lCGLNyqVSGJYJVuOlKRuDNGmjS6RZpnXO3jZyDi2tMgxjDJW8nBB514QWYG3UHVnGpSywxGI0tAl2L56+6uWxJHYNGqPJl4KGQBJfifExvqm1fOH9d69TuFqqy0x05cogo1yvPosZ8gbAhIK3ssNTouDFgUqFU8FvMlC1qzUcjzYTL11c/XcMQ== Authentication-Results: fb.com; dkim=none (message not signed) header.d=none;fb.com; dmarc=none action=none header.from=nvidia.com; Received: from MN2PR12MB3823.namprd12.prod.outlook.com (2603:10b6:208:168::26) by MN2PR12MB4437.namprd12.prod.outlook.com (2603:10b6:208:26f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Wed, 31 Mar 2021 02:04:20 +0000 Received: from MN2PR12MB3823.namprd12.prod.outlook.com ([fe80::a1b1:5d8:47d7:4b60]) by MN2PR12MB3823.namprd12.prod.outlook.com ([fe80::a1b1:5d8:47d7:4b60%7]) with mapi id 15.20.3977.033; Wed, 31 Mar 2021 02:04:20 +0000 From: "Zi Yan" To: "Roman Gushchin" Cc: linux-mm@kvack.org, "Matthew Wilcox" , "Kirill A . Shutemov" , "Andrew Morton" , "Yang Shi" , "Michal Hocko" , "John Hubbard" , "Ralph Campbell" , "David Nellans" , "Jason Gunthorpe" , "David Rientjes" , "Vlastimil Babka" , "David Hildenbrand" , "Mike Kravetz" , "Song Liu" Subject: Re: [RFC PATCH v3 00/49] 1GB PUD THP support on x86_64 Date: Tue, 30 Mar 2021 22:04:13 -0400 X-Mailer: MailMate (1.14r5757) Message-ID: In-Reply-To: References: <20210224223536.803765-1-zi.yan@sent.com> <890DE8FE-DAF6-49A2-8C62-40B6FD593B4A@nvidia.com> <06D1034A-DE8B-4970-9056-6CA1C436D2E8@nvidia.com> Content-Type: multipart/signed; boundary="=_MailMate_48EC6B31-EB8E-4066-B010-3BBAC9EC77BA_="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Originating-IP: [216.228.112.21] X-ClientProxiedBy: MN2PR02CA0001.namprd02.prod.outlook.com (2603:10b6:208:fc::14) To MN2PR12MB3823.namprd12.prod.outlook.com (2603:10b6:208:168::26) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [10.2.57.140] (216.228.112.21) by MN2PR02CA0001.namprd02.prod.outlook.com (2603:10b6:208:fc::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25 via Frontend Transport; Wed, 31 Mar 2021 02:04:15 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: beb3a36d-22a6-43b5-c707-08d8f3e94bbb X-MS-TrafficTypeDiagnostic: MN2PR12MB4437: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oiL9EbxdPGkVxdE7E/X8dWMl76eDXs+NFiwpODrh8Uq7qrh/0tVQ1Hfqqufoc2K1ofDxcdyHxxPlds8fxmx0hVNtHGOdza0/RTLCIOHarhEgqHPW3q7nUHRW25/dNOAdYZMOjtjUa+sCjiIf2lOC49ViFxdUjp1kdpKml8oIjw7q+zNl5l+yGwPmhVqh79VEV90M5UPX19PpYLGEwB7G21uopqIcC+JSBHVq6sos5wDciQPOvOWS3pKo26yROSbF9FeoeQ2bhj8Sk0iw646LDVQs362wjPyJiHfYxIkxzEOyGMzwZy0ZABgOMBkYz/X2Vq1wBauZLiCurfb5jL3sdESgV2q9vo0ZTVZdz/ThMhx+z7UFN9w26LesJd0BfYSufcOziUWhzUKF54Dk7PEc+pjb/UQnIVFfmSs3rjeGD1ZlRSrLYajxDMb08op8ASkb+n3e9PqqIT+Im+OojN3W708gzkxo+yFI/wu/yoWSem23lpeRgJWS+jYSZGoUsnExHnFOHsPG3gBklH2ouwG6FIP1F+50tqifIQJDoDwWNtdnIhjKpjpNRJSEZT/TA5xkaY5x3lT84glf/QoUgkkosEP2ancZsQTfK/e5x8VgeFxfpbTJ/7LL84bezWLpaJz1mWtmoRj8X4qhx0TGuxkG+WlCjtIKnipw113Jft/GDXMG6xdmTzyFLfTvK7YETycc+GkFQuSDXacLIqafaK9A3uVGtKC291WhCe1rwESOVxWJVa1sHRY0um+5E7LIZsa9vxJ1xMIRHJCc5dvRUEhDPXxiekqwaKPK8RlzQ7fi+OY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR12MB3823.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(346002)(136003)(396003)(376002)(39860400002)(366004)(8936002)(33656002)(45080400002)(7416002)(6486002)(186003)(6916009)(966005)(8676002)(33964004)(53546011)(4326008)(2906002)(5660300002)(235185007)(26005)(38100700001)(478600001)(2616005)(16526019)(956004)(86362001)(66946007)(66476007)(66556008)(16576012)(54906003)(316002)(83380400001)(36756003)(72826003)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?MHk0SXNJV0pZQVJ6TGJnYmVUQ0FnR2VENlR5dFFmeUZYWFpzblRUS2tsRWp1?= =?utf-8?B?THRsSG9kTFd4ZzBDSC9FNExwRTlUQjNtZnY1Zm40UlFWaUY1YS9pQzFJdlhY?= =?utf-8?B?L3dQSFZkMWQwaVdJRjZuS2FkWkxzL28vbmxraFZNenczWWh3VGxsV0U2V2N1?= =?utf-8?B?cXB0OEtGRk9QaTVJSmxiTTRqVWRFdEZZS3Bjb3UwelNreGlyNmVwUEtVRHlO?= =?utf-8?B?QTI4V1JPRTFUUTI0K1FuWjZSWkRYSGhqWWpIOXhncThrajRHbjUySC8yTjVZ?= =?utf-8?B?SjRraERDbC95eTh4dWZjTi9oYy84S3R1YXZzUm4zZ1VRdWMrUE5hT3FtSFpo?= =?utf-8?B?U2Q5eVNFdFJSTllWSWhxNWdtOTQ3bmdINDlDZnVzY05XUUtDblRBZzlIYzRR?= =?utf-8?B?WmRLR3ZIcVRrWEtaU2xrd0FZTmtXeXFkeHNoYjlXUXJLMFNTRTdHeFR1RXZr?= =?utf-8?B?YXVJc3Y5dHVmSWJqS3RxbG5BVzQyODZuZS96ajRGQU11ZGZ5R01xQlprL05K?= =?utf-8?B?aldtcWlEYUlZTDZYQWt3WVNnQVpSNFZQellJbWkwVjhWaDFRNXR1SlJoclhO?= =?utf-8?B?cWdxTS9UdXFUUGdqTzNYeEZ0aFY3emFVZVIzMmg0Mm9LNHdCQXdQdlpUWVNq?= =?utf-8?B?TzJ3eXhiQkRoU0g5dm1EdnRsMFBEcGFsbGwxYWo2SXpPMW1raFBUb2JOMXpQ?= =?utf-8?B?S2xjUGtJNlJ1ZWkwYXlSd0FuN1E3NncrNEhydXFMeFpxQTNIeDNDZFAwTHow?= =?utf-8?B?V1pnZm5tdG1jd2I5TCtlMDcvTC9MdENlQ3lUYkdmTTJtWlc3M0JXejRpN2ZF?= =?utf-8?B?UDVRei9VK2tTU3BXU1RZd0lmbk8vb25OM28zRUd1OFgxa292VXdOZUt2TjZx?= =?utf-8?B?WFAzUVdwVnVwWmN2eHpHZktUVWZLZnJteFZsTGRseEMrcUwrWmptN01XVGtY?= =?utf-8?B?d0ZvYkxyYS8wSFYwZC9RczE2ck5lMkNkODlLdDBSbTg5VjJuM1RxME9XQVoz?= =?utf-8?B?cTEzaGYwelRlN0hsNUdxRjl4MzhqL1lrN0VMR1g0ekJIeWZ4VFNndjVLVmE4?= =?utf-8?B?MEtwYzVzMXZQNnBUbnRvU3o5dExhZk8zbkRVcDhTWXdFODlrZW84SzlqSnVX?= =?utf-8?B?NDU4d0JncWFhNEN3NHpwNEtzR0RXbEFhZGxBdTFUTHNkTmMydDRXUkRNN3Rj?= =?utf-8?B?WXNWWTNDZHhKdDRwR0dyS2lEdmgvMHlRZHJWcy9iSnBTYWl6ZlhxRSt5amE4?= =?utf-8?B?cExyeWx1aWZ2aTZ3aFpwdXo5Q2VqK3RncVM0L2s0TGFGc3czd3hUT2ZWYTk4?= =?utf-8?B?NEZaSzJTQXhBb1BUL3Azc1l6K1UrdmVZUHk4aWpCT3dlelRWMk8zaEJhMnRo?= =?utf-8?B?MWRnWTlWNEhCYkQrSE40TEw3ZTVxaURFb1BxK2x0TTRTT0xlNHdxazBkWGlO?= =?utf-8?B?WTZLSi9mazU2T1NHTzdrOXpBblI2dUJsVWpyZDl4SS9qcjJaaFVyUzEwYyt6?= =?utf-8?B?S3Z2S21oUFVHcnlROHNobGlZeGY3Vjk3QXhZbnl6TWZZazcyK3pRQVZncDV3?= =?utf-8?B?TkZleW85anJPTms0d3lUbEpsbExRR0dJOHJMbEtvL25XVnh2aHFvZEdCUUww?= =?utf-8?B?N3dtV0gzL29YTjBPcUUzMkdpK2xTYUt3ajY0QnhUTnk5UTRoeGR0WUR5SG9t?= =?utf-8?B?NjEwYjVETWdNN2tiaElKZ3FvVVM0NWRXSHdqbktJYzQvcTh4dGNob3AvVHpT?= =?utf-8?Q?dTAN5vK0dvt4N+dTV1zqu7DzFiBf5PDVQIlwhk0?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: beb3a36d-22a6-43b5-c707-08d8f3e94bbb X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3823.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Mar 2021 02:04:19.9570 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: msBNCCk9LmFENgavJIg5NbZ89j9BDlMnyNvyh3lOn0G9DVnAJa6tfTvzZ67IksS5 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4437 X-Rspamd-Queue-Id: E1D7940002CF X-Stat-Signature: 1r14i9y7516n68uhdadjck7jo56g4afs X-Rspamd-Server: rspam02 Received-SPF: none (nvidia.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=NAM11-BN8-obe.outbound.protection.outlook.com; client-ip=40.107.236.66 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617156263-89515 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --=_MailMate_48EC6B31-EB8E-4066-B010-3BBAC9EC77BA_= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On 30 Mar 2021, at 14:02, Roman Gushchin wrote: > On Tue, Mar 30, 2021 at 01:24:14PM -0400, Zi Yan wrote: >> Hi Roman, >> >> >> On 4 Mar 2021, at 11:45, Roman Gushchin wrote: >> >>> On Thu, Mar 04, 2021 at 11:26:03AM -0500, Zi Yan wrote: >>>> On 1 Mar 2021, at 20:59, Roman Gushchin wrote: >>>> >>>>> On Wed, Feb 24, 2021 at 05:35:36PM -0500, Zi Yan wrote: >>>>>> From: Zi Yan >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I have rebased my 1GB PUD THP support patches on v5.11-mmotm-2021-= 02-18-18-29 >>>>>> and the code is available at >>>>>> https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%= 2Fgithub.com%2Fx-y-z%2Flinux-1gb-thp%2Ftree%2F1gb_thp_v5.11-mmotm-2021-02= -18-18-29&data=3D04%7C01%7Cziy%40nvidia.com%7C49dd8b5e66994e6b7f5e08d= 8f3a5fa13%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637527241503834147= %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h= aWwiLCJXVCI6Mn0%3D%7C1000&sdata=3D3jjPz8HTJDn3bYWhrwKToMCXDScZuCoqsEs= ink3eGZE%3D&reserved=3D0 >>>>>> if you want to give it a try. The actual 49 patches are not sent o= ut with this >>>>>> cover letter. :) >>>>>> >>>>>> Instead of asking for code review, I would like to discuss on the = concerns I got >>>>>> from previous RFCs. I think there are two major ones: >>>>>> >>>>>> 1. 1GB page allocation. Current implementation allocates 1GB pages= from CMA >>>>>> regions that are reserved at boot time like hugetlbfs. The conc= erns on >>>>>> using CMA is that an educated guess is needed to avoid depletin= g kernel >>>>>> memory in case CMA regions are set too large. Recently David Ri= entjes >>>>>> proposes to use process_madvise() for hugepage collapse, which = is an >>>>>> alternative [1] but might not work for 1GB pages, since there i= s no way of >>>>>> _allocating_ a 1GB page to which collapse pages. I proposed a s= imilar >>>>>> approach at LSF/MM 2019, generating physically contiguous memor= y after pages >>>>>> are allocated [2], which is usable for 1GB THPs. This approach = does in-place >>>>>> huge page promotion thus does not require page allocation. >>>>> >>>>> Well, I don't think there an alternative to cma as now. When the me= mory is almost >>>>> filled at least once, any subsequent activity leading to substantia= l slab allocations >>>>> (e.g. run git gc) will fragment the memory, so that there are virtu= ally no chances >>>>> to find a continuous GB. >>>>> >>>>> It's possible in theory to reduce the fragmentation on 1GB scale by= grouping >>>>> non-movable pageblocks, but it seems a separate project. >>>> >>>> My experiments showed that finding continuous GBs is possible, but I= agree that >>>> CMA is more reliable and 1GB scale defragmentation should be a separ= ate project. >>> >>> I actually ran a large scale experiment (on tens of thousands of mach= ines) in the last >>> several months. It was about hugetlbfs 1GB pages, but the allocation = mechanism is the same. >> >> Thanks for the information. I finally have time to come back to this. = Do you mind sharing >> the total memory of these machines? I want to have some idea on the sc= ale of this issue to >> make sure I reproduce in a proper machine. Are you trying to get <20% = of 10s GBs, 100s GBs, >> or TBs memory? > > There are different configurations, but in general they are in 100's GB= or smaller. > >> >>> >>> My goal as to allocate a relatively small number of 1GB pages (<20% o= f the total memory). >>> Without cma chances are reaching 0% very fast after reboot, and even = manual manipulations >>> like shutting down all workloads, dropping caches, calling sync, comp= action, etc. do not >>> help much. Sometimes you can allocate maybe 1-2 pages, but that's abo= ut it. >> >> Is there a way of replicating such an environment with publicly availa= ble software? >> I really want to understand the root cause and am willing to find a po= ssible solution. >> It would be much easier if I can reproduce this locally. > > There is nothing fb-specific: once the memory is filled with anon/pagec= ache, any subsequent > allocations of non-movable memory (slabs, percpu, etc) will fragment th= e memory. There > is a pageblock mechanism which prevents the fragmentation on 2MB scale,= but nothing prevents > the fragmentation on 1GB scale. It just a matter of runtime (and the nu= mber of mm operations). > >> >>> >>> Even with cma we had to fix a number of additional problems (like sub= -optimal placement >>> of cma areas, 2MB THP migration, some ext4 and btrfs page migration i= ssues) to have >>> a reasonable success rate about ~95-99%. And it's not 100% anyway. >>> >>> The problem with artificial tests is that you're likely experimenting= on a freshly >>> rebooted machine which isn't/wasn't doing much. It's a bad model of t= he real memory >>> state of a production server. >> >> Yes, I agree that my experiment is not representative. Can you provide= more information >> on what application behavior(s) leading to this memory fragmentation? = I guess it is >> because non-moveable pages spread across the entire physical memory sp= ace. Is there >> a quick reproducer for that? > > I have a simple c program which is able to fragment the memory, you can= play with it: > https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgit= hub.com%2Frgushchin%2Ffragm&data=3D04%7C01%7Cziy%40nvidia.com%7C49dd8= b5e66994e6b7f5e08d8f3a5fa13%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C= 637527241503834147%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2= luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DRE9CfPG2fG7lZfHui= W78jlJewajJzJ2DCbbmGJpWPRU%3D&reserved=3D0 . > > But as I said, basically any load which is actively using the whole mem= ory > will fragment it. With your simple program, I am able to fragment the memory to the conditi= on that it is impossible to allocate/generate 1GB pages. I will look into this. Thanks. =E2=80=94 Best Regards, Yan Zi --=_MailMate_48EC6B31-EB8E-4066-B010-3BBAC9EC77BA_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJDBAEBCgAtFiEEh7yFAW3gwjwQ4C9anbJR82th+ooFAmBj2J0PHHppeUBudmlk aWEuY29tAAoJEJ2yUfNrYfqKwQoP/0DnodaOi29WJWAZNeTEMpGhFe7xwOZdYoU+ GPExlKEPuyKnlOc63vPMCzubMugpiOcvAQHVGZsgj/wl6UWqhvli2nuk13wTkD44 +ne3ay2ojLRVRoEyHPMtTp+IPXwxQi8qKL+IYh8QpR7kAcH9LwtpMoEgQc0xNyBC dEEB2G4myYWFpugVh8DXC/ncguS+vmD0ozER0DUpIDYcEUixPEYw4bnJdAX8ysZ/ lRRq70Dzg/tDI27k8OE6XpFgHyCXYjy5MpKYoDf16ZGAs7tRlf4B36aDIWjr387N 9gbIHVnfvqsz/FWe/rZFuLdpaDSS5FHGnQ3FppPLhXwiV1a4w9absh/xqXHGnybp sC3PAWG2yDZ0VZzXAaspx30jBJE3Kj4yyEDuVFF+Zr6tAhgv17FRggM465jRI+MR AxhkhDd+2SrROP2cytrHwq9VEvpxgBRpEIexzQTNjlQ23WyMibz5ec/hSTqix6vu PoufK5T7ep5iFabI4KU48DTaoz8Y+MSC9dCRsl49UMTflqSu/KdKtPXuiyBEAKwP 42PycF3RMiyKgd2j3UYwPNu4vt0yBplMMnrHX5p+vp4xWUoslOwXY5mXtsfhCys6 TxbRzcWm/n7P3sAC69ipaLdUd/EUz2lDzKF2jiednPGspZD05VujAZklXsqTUnqc XDCyocJb =I4+L -----END PGP SIGNATURE----- --=_MailMate_48EC6B31-EB8E-4066-B010-3BBAC9EC77BA_=--