From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8614ECAAD8 for ; Wed, 21 Sep 2022 06:06:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEAE26B0073; Wed, 21 Sep 2022 02:06:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AAE8B6B0074; Wed, 21 Sep 2022 02:06:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 976E3940007; Wed, 21 Sep 2022 02:06:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8995E6B0073 for ; Wed, 21 Sep 2022 02:06:55 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 449C4140F18 for ; Wed, 21 Sep 2022 06:06:55 +0000 (UTC) X-FDA: 79935059190.24.20BCED8 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf02.hostedemail.com (Postfix) with ESMTP id AD1AF8000E for ; Wed, 21 Sep 2022 06:06:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663740414; x=1695276414; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P89pEx/QWVFzErE/u067pCGTE2O1BWblDLS8JJgMTCo=; b=NPJc4zWXq4Rke5uHzHcvH4SzrfzfqSxOfxo90ark/n/jfd6xej8QmD7W 9ooj7PYwJHcxQOwu6HTKh+0duLC2RwO1rgX/Wt/SVtephPcKnn+SFR68S iMPg2miGiW3QLeNpqNvDIhaH/t0qaTrJ04QfiLvizt9Zw80GcoNISO3N2 8Kx8rUZmVcdXBc5n3r9Ndc8pXAEuNxvhVRu0q0HKibkxNjDTV4EXDRz2q k3tg7YgLV0UJMX8CrMnGrmDy1fcz9PALMUY5kxQMN0Zkti9K38lvZ6EkI qWGIQ2xsMtMHToip/ruYlvwZ1F3NsgSxNYU5cjP3wMXBIxxPJqHJjjrBI w==; X-IronPort-AV: E=McAfee;i="6500,9779,10476"; a="282956817" X-IronPort-AV: E=Sophos;i="5.93,332,1654585200"; d="scan'208";a="282956817" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Sep 2022 23:06:53 -0700 X-IronPort-AV: E=Sophos;i="5.93,332,1654585200"; d="scan'208";a="649913842" Received: from yhuang6-mobl2.sh.intel.com ([10.238.5.245]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Sep 2022 23:06:50 -0700 From: Huang Ying To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox Subject: [RFC 1/6] mm/migrate_pages: separate huge page and normal pages migration Date: Wed, 21 Sep 2022 14:06:11 +0800 Message-Id: <20220921060616.73086-2-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220921060616.73086-1-ying.huang@intel.com> References: <20220921060616.73086-1-ying.huang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663740414; a=rsa-sha256; cv=none; b=g4sNiuw5wuULQrjIP1gIoWoejf2LYMcsHoIaouF6FcsuqYS1I1fKeO8NZ+xUI3Ty35gI1W yAxJPcxPB+QwVaFJ+yThdulkrnQGcI3HW0YOlp/O3+dEp+s2qiUHu/FzVHxLEmK7Lz8FoV CL6rOry7+FimovEE14JcuhmHyKPj/TQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=NPJc4zWX; spf=pass (imf02.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663740414; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iEWjYvdzXc766o9gXUnYzm/lZBr/zsno7aHFZjyF2lk=; b=R05rPvL2TEFS5n11up952uhSI6wajFTnppMkMOJS+NJD5tUD9QZG/f3bzkiU59A3t7Afu6 dAav4pfZXso3TCpOE2EttuLBJ5vijGLqvbi+VmFQOvMb+wKwpCfpzYQvHsYznU2/McsEJo kDurQ9Hr/zlLHPgI8FVOr4GemjgUjk0= X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AD1AF8000E Authentication-Results: imf02.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=NPJc4zWX; spf=pass (imf02.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: u4eaq3esucgp4ysbhz8g5nr1w4w374u7 X-HE-Tag: 1663740414-619721 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying. In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox --- mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 64 insertions(+), 9 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 571d8c9fd5bc..117134f1c6dc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); + for (pass = 0; pass < 10 && retry; pass++) { + retry = 0; + + list_for_each_entry_safe(page, page2, from, lru) { + nr_subpages = compound_nr(page); + cond_resched(); + + if (!PageHuge(page)) + continue; + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, page, + pass > 2, mode, reason, + &ret_pages); + /* + * The rules are: + * Success: hugetlb page will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_pages list then splice to + * from list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + nr_failed_pages += nr_subpages; + list_move_tail(&page->lru, &ret_pages); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other pages, just exit. + */ + nr_failed++; + nr_failed_pages += nr_subpages + nr_retry_pages; + goto out; + case -EAGAIN: + retry++; + nr_retry_pages += nr_subpages; + break; + case MIGRATEPAGE_SUCCESS: + nr_succeeded += nr_subpages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed page is + * removed from migration page list and not + * retried in the next outer loop. + */ + nr_failed++; + nr_failed_pages += nr_subpages; + break; + } + } + } + nr_failed += retry; + retry = 1; thp_subpage_migration: for (pass = 0; pass < 10 && (retry || thp_retry); pass++) { retry = 0; @@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, cond_resched(); if (PageHuge(page)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, page, - pass > 2, mode, reason, - &ret_pages); - else - rc = unmap_and_move(get_new_page, put_new_page, + continue; + + rc = unmap_and_move(get_new_page, put_new_page, private, page, pass > 2, mode, reason, &ret_pages); /* * The rules are: - * Success: non hugetlb page will be freed, hugetlb - * page will be put back + * Success: page will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_thp_split++; break; } - /* Hugetlb migration is unsupported */ } else if (!no_subpage_counting) { nr_failed++; } -- 2.35.1