On 4 Jun 2020, at 23:35, Anshuman Khandual wrote: > On 06/04/2020 10:19 PM, Zi Yan wrote: >> On 4 Jun 2020, at 12:36, Matthew Wilcox wrote: >> >>> On Thu, Jun 04, 2020 at 09:51:10AM -0400, Zi Yan wrote: >>>> On 4 Jun 2020, at 7:34, Matthew Wilcox wrote: >>>>> On Thu, Jun 04, 2020 at 09:30:45AM +0530, Anshuman Khandual wrote: >>>>>> +Quantifying Migration >>>>>> +===================== >>>>>> +Following events can be used to quantify page migration. >>>>>> + >>>>>> +- PGMIGRATE_SUCCESS >>>>>> +- PGMIGRATE_FAIL >>>>>> +- THP_MIGRATION_SUCCESS >>>>>> +- THP_MIGRATION_FAILURE >>>>>> + >>>>>> +THP_MIGRATION_FAILURE in particular represents an event when a THP could not be >>>>>> +migrated as a single entity following an allocation failure and ended up getting >>>>>> +split into constituent normal pages before being retried. This event, along with >>>>>> +PGMIGRATE_SUCCESS and PGMIGRATE_FAIL will help in quantifying and analyzing THP >>>>>> +migration events including both success and failure cases. >>>>> >>>>> First, I'd suggest running this paragraph through 'fmt'. That way you >>>>> don't have to care about line lengths. >>>>> >>>>> Second, this paragraph doesn't really explain what I need to know to >>>>> understand the meaning of these numbers. When Linux attempts to migrate >>>>> a THP, one of three things can happen: >>>>> >>>>> - It is migrated as a single THP >>>>> - It is migrated, but had to be split >>>>> - Migration fails >>>>> >>>>> How do I turn these four numbers into an understanding of how often each >>>>> of those three situations happen? And why do we need four numbers to >>>>> report three situations? >>>>> >>>>> Or is there something else that can happen? If so, I'd like that explained >>>>> here too ;-) >>>> >>>> PGMIGRATE_SUCCESS and PGMIGRATE_FAIL record a combination of different events, >>>> so it is not easy to interpret them. Let me try to explain them. >>> >>> Thanks! Very helpful explanation. >>> >>>> 1. migrating only base pages: PGMIGRATE_SUCCESS and PGMIGRATE_FAIL just mean >>>> these base pages are migrated and fail to migrate respectively. >>>> THP_MIGRATION_SUCCESS and THP_MIGRATION_FAILURE should be 0 in this case. >>>> Simple. >>>> >>>> 2. migrating only THPs: >>>> - PGMIGRATE_SUCCESS means THPs that are migrated and base pages >>>> (from the split of THPs) that are migrated, >>>> >>>> - PGMIGRATE_FAIL means THPs that fail to migrate and base pages that fail to migrated. >>>> >>>> - THP_MIGRATION_SUCCESS means THPs that are migrated. >>>> >>>> - THP_MIGRATION_FAILURE means THPs that are split. >>>> >>>> So PGMIGRATE_SUCCESS - THP_MIGRATION_SUCCESS means the number of migrated base pages, >>>> which are from the split of THPs. >>> >>> Are you sure about that? If I split a THP and each of those subpages >>> migrates, won't I then see PGMIGRATE_SUCCESS increase by 512? >> >> That is what I mean. I guess my words did not work. I should have used subpages. >> >>> >>>> When it comes to analyze failed migration, PGMIGRATE_FAIL - THP_MIGRATION_FAILURE >>>> means the number of pages that are failed to migrate, but we cannot tell how many >>>> are base pages and how many are THPs. >>>> >>>> 3. migrating base pages and THP: >>>> >>>> The math should be very similar to the second case, except that >>>> a) from PGMIGRATE_SUCCESS - THP_MIGRATION_SUCCESS, we cannot tell how many are pages begin >>>> as base pages and how many are pages begin as THPs but become base pages after split; >>>> b) from PGMIGRATE_FAIL - THP_MIGRATION_FAILURE, an additional case, >>>> base pages that begin as base pages fail to migrate, is mixed into the number and we >>>> cannot tell three cases apart. >>> >>> So why don't we just expose PGMIGRATE_SPLIT? That would be defined as >>> the number of times we succeeded in migrating a THP but had to split it >>> to succeed. >> >> It might need extra code to get that number. Currently, the subpages from split >> THPs are appended to the end of the original page list, so we might need a separate >> page list for these subpages to count PGMIGRATE_SPLIT. Also what if some of the >> subpages fail to migrate? Do we increase PGMIGRATE_SPLIT or not? > > Thanks Zi, for such a detailed explanation. Ideally, we should separate THP > migration from base page migration in terms of statistics. PGMIGRATE_SUCCESS > and PGMIGRATE_FAIL should continue to track statistics when migration starts > with base pages. But for THP, we should track the following events. You mean PGMIGRATE_SUCCESS and PGMIGRATE_FAIL will not track the number of migrated subpages from split THPs? Will it cause userspace issues since their semantics are changed? > > 1. THP_MIGRATION_SUCCESS - THP migration is successful, without split > 2. THP_MIGRATION_FAILURE - THP could neither be migrated, nor be split They make sense to me. > 3. THP_MIGRATION_SPLIT_SUCCESS - THP got split and all sub pages migrated > 4. THP_MIGRATION_SPLIT_FAILURE - THP got split but all sub pages could not be migrated > > THP_MIGRATION_SPLIT_FAILURE could either increment once for a single THP or > number of subpages that did not get migrated after split. As you mentioned, > this will need some extra code in the core migration. Nonetheless, if these > new events look good, will be happy to make required changes. Maybe THP_MIGRATION_SPLIT would be simpler? My concern is that whether we need such detailed information or not. Maybe trace points would be good enough for 3 and 4. But if you think it is useful to you, feel free to implement them. BTW, in terms of stats tracking, what do you think of my patch below? I am trying to aggregate all stats counting in one place. Feel free to use it if you think it works for you. diff --git a/mm/migrate.c b/mm/migrate.c index 7bfd0962149e..0f3c60470489 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1429,9 +1429,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason) { int retry = 1; + int thp_retry = 1; int nr_failed = 0; + int nr_thp_failed = 0; + int nr_thp_split = 0; int nr_succeeded = 0; + int nr_thp_succeeded = 0; int pass = 0; + bool is_thp = false; struct page *page; struct page *page2; int swapwrite = current->flags & PF_SWAPWRITE; @@ -1440,11 +1445,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (!swapwrite) current->flags |= PF_SWAPWRITE; - for(pass = 0; pass < 10 && retry; pass++) { + for(pass = 0; pass < 10 && (retry || thp_retry); pass++) { retry = 0; + thp_retry = 0; list_for_each_entry_safe(page, page2, from, lru) { retry: + is_thp = PageTransHuge(page); cond_resched(); if (PageHuge(page)) @@ -1475,15 +1482,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, unlock_page(page); if (!rc) { list_safe_reset_next(page, page2, lru); + nr_thp_split++; goto retry; } } nr_failed++; goto out; case -EAGAIN: + if (is_thp) + thp_retry++; retry++; break; case MIGRATEPAGE_SUCCESS: + if (is_thp) + nr_thp_succeeded++; nr_succeeded++; break; default: @@ -1493,18 +1505,27 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * removed from migration page list and not * retried in the next outer loop. */ + if (is_thp) + nr_thp_failed++; nr_failed++; break; } } } nr_failed += retry; + nr_thp_failed += thp_retry; rc = nr_failed; out: if (nr_succeeded) count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded); if (nr_failed) count_vm_events(PGMIGRATE_FAIL, nr_failed); + if (nr_thp_succeeded) + count_vm_events(THP_MIGRATION_SUCCESS, nr_thp_succeeded); + if (nr_thp_failed) + count_vm_events(THP_MIGRATION_FAIL, nr_thp_failed); + if (nr_thp_split) + count_vm_events(THP_MIGRATION_SPLIT, nr_thp_split); trace_mm_migrate_pages(nr_succeeded, nr_failed, mode, reason); if (!swapwrite) — Best Regards, Yan Zi