On 12 Apr 2022, at 10:49, David Hildenbrand wrote: > On 12.04.22 16:07, Zi Yan wrote: >> On 12 Apr 2022, at 9:10, David Hildenbrand wrote: >> >>> On 06.04.22 17:18, Zi Yan wrote: >>>> From: Zi Yan >>>> >>>> Enable set_migratetype_isolate() to check specified sub-range for >>>> unmovable pages during isolation. Page isolation is done >>>> at MAX_ORDER_NR_PAEGS granularity, but not all pages within that >>>> granularity are intended to be isolated. For example, >>>> alloc_contig_range(), which uses page isolation, allows ranges without >>>> alignment. This commit makes unmovable page check only look for >>>> interesting pages, so that page isolation can succeed for any >>>> non-overlapping ranges. >>>> >>>> Signed-off-by: Zi Yan >>>> --- >>> >>> [...] >>> >>>> /* >>>> - * This function checks whether pageblock includes unmovable pages or not. >>>> + * This function checks whether the range [start_pfn, end_pfn) includes >>>> + * unmovable pages or not. The range must fall into a single pageblock and >>>> + * consequently belong to a single zone. >>>> * >>>> * PageLRU check without isolation or lru_lock could race so that >>>> * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable >>>> @@ -28,12 +30,14 @@ >>>> * cannot get removed (e.g., via memory unplug) concurrently. >>>> * >>>> */ >>>> -static struct page *has_unmovable_pages(struct zone *zone, struct page *page, >>>> - int migratetype, int flags) >>>> +static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long end_pfn, >>>> + int migratetype, int flags) >>>> { >>>> - unsigned long iter = 0; >>>> - unsigned long pfn = page_to_pfn(page); >>>> - unsigned long offset = pfn % pageblock_nr_pages; >>>> + unsigned long pfn = start_pfn; >>>> + struct page *page = pfn_to_page(pfn); >>> >>> >>> Just do >>> >>> struct page *page = pfn_to_page(start_pfn); >>> struct zone *zone = page_zone(page); >>> >>> here. No need to lookup the zone again in the loop because, as you >>> document "must ... belong to a single zone.". >>> >>> Then, there is also no need to initialize "pfn" here. In the loop header >>> is sufficient. >>> >> >> Sure. >> >>>> + >>>> + VM_BUG_ON(ALIGN_DOWN(start_pfn, pageblock_nr_pages) != >>>> + ALIGN_DOWN(end_pfn - 1, pageblock_nr_pages)); >>>> >>>> if (is_migrate_cma_page(page)) { >>>> /* >>>> @@ -47,8 +51,11 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, >>>> return page; >>>> } >>>> >>>> - for (; iter < pageblock_nr_pages - offset; iter++) { >>>> - page = pfn_to_page(pfn + iter); >>>> + for (pfn = start_pfn; pfn < end_pfn; pfn++) { >>>> + struct zone *zone; >>>> + >>>> + page = pfn_to_page(pfn); >>>> + zone = page_zone(page); >>>> >>>> /* >>>> * Both, bootmem allocations and memory holes are marked >>>> @@ -85,7 +92,7 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, >>>> } >>>> >>>> skip_pages = compound_nr(head) - (page - head); >>>> - iter += skip_pages - 1; >>>> + pfn += skip_pages - 1; >>>> continue; >>>> } >>>> >>>> @@ -97,7 +104,7 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, >>>> */ >>>> if (!page_ref_count(page)) { >>>> if (PageBuddy(page)) >>>> - iter += (1 << buddy_order(page)) - 1; >>>> + pfn += (1 << buddy_order(page)) - 1; >>>> continue; >>>> } >>>> >>>> @@ -134,11 +141,18 @@ static struct page *has_unmovable_pages(struct zone *zone, struct page *page, >>>> return NULL; >>>> } >>>> >>>> -static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags) >>>> +/* >>>> + * This function set pageblock migratetype to isolate if no unmovable page is >>>> + * present in [start_pfn, end_pfn). The pageblock must intersect with >>>> + * [start_pfn, end_pfn). >>>> + */ >>>> +static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags, >>>> + unsigned long start_pfn, unsigned long end_pfn) >>> >>> I think we might be able do better, eventually not passing start_pfn at >>> all. Hmm. >> >> IMHO, having start_pfn and end_pfn in the parameter list would make the >> interface easier to understand. Otherwise if we remove start_pfn, >> the caller needs to adjust @page to be within the range of [start_pfn, >> end_pfn) >> >>> >>> I think we want to pull out the >>> start_isolate_page_range()/undo_isolate_page_range() interface change >>> into a separate patch. >> >> You mean a patch just adding >> >> unsigned long isolate_start = pfn_max_align_down(start_pfn); >> unsigned long isolate_end = pfn_max_align_up(end_pfn); >> >> in start_isolate_page_range()/undo_isolate_page_range()? >> >> Yes I can do that. > > I think we have to be careful with memory onlining/offlining. There are > corner cases where we get called with only pageblock alignment and > must not adjust the range. In the patch below, you added a new set of start_isolate_pageblocks() and undo_isolate_pageblocks(), where start_isolate_pageblocks() still calls set_migratetype_isolate() and noted their range should not be adjusted. But in my patch, set_migratetype_isolate() adjusts the range for has_unmovable_pages() check. Do you mean start_isolate_pageblocks() should call a different version of set_migratetype_isolate() without range adjustment? That can be done with an additional parameter in start_isolate_page_range(), like bool strict, right? > > > Something like this as a base for the next cleanups/extensions: > > > From 18d29b53600d6d0d6ac87cdc6b7517e989fa3dac Mon Sep 17 00:00:00 2001 > From: David Hildenbrand > Date: Tue, 12 Apr 2022 15:51:50 +0200 > Subject: [PATCH] mm: page-isolation: Move alignment logic into > start_isolate_page_range()/undo_isolate_page_range() > > For ordinary range allocations, we actually have to isolate all pageblocks > in a MAX_ORDER - 1 range. Only memory onlining/offlining is special: it > knows exactly which pageblocks to isolate/unisolate and we must not mess > with the pageblocks to isolate (memory onlining/offlining alwayes passed > MAX_ORDER - 1 - aligned ranges, unless we're dealing with vmemmap > located on hotplugged memory, whereby the start pfn might only be > pageblock aligned). > > Further, for ordinary allcoations, we'll want to know the exact range > we want to allocate -- to check only that range for unmovable pages. > Right now we lose that information. > > So let's move the alignment logic into start_isolate_page_range() / > undo_isolate_page_range(), such that we have the actual range of > interest available and the alignment logic contained in there. > > Provide start_isolate_pageblocks()/undo_isolate_pageblocks() for memory > onlining/offlining. > > Signed-off-by: David Hildenbrand > --- > include/linux/page-isolation.h | 23 ++++----- > mm/memory_hotplug.c | 8 ++-- > mm/page_alloc.c | 15 +----- > mm/page_isolation.c | 85 ++++++++++++++++++++++++++-------- > 4 files changed, 81 insertions(+), 50 deletions(-) > > diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h > index e14eddf6741a..8e9e9e80ba67 100644 > --- a/include/linux/page-isolation.h > +++ b/include/linux/page-isolation.h > @@ -37,20 +37,15 @@ void set_pageblock_migratetype(struct page *page, int migratetype); > int move_freepages_block(struct zone *zone, struct page *page, > int migratetype, int *num_movable); > > -/* > - * Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE. > - */ > -int > -start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > - unsigned migratetype, int flags); > - > -/* > - * Changes MIGRATE_ISOLATE to MIGRATE_MOVABLE. > - * target range is [start_pfn, end_pfn) > - */ > -void > -undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > - unsigned migratetype); > +int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype, int flags); > +int start_isolate_pageblocks(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype, int flags); > + > +void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype); > +void undo_isolate_pageblocks(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype); > > /* > * Test all pages in [start_pfn, end_pfn) are isolated or not. > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c > index 416b38ca8def..fb7f63c800d1 100644 > --- a/mm/memory_hotplug.c > +++ b/mm/memory_hotplug.c > @@ -1089,7 +1089,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, > > /* > * Fixup the number of isolated pageblocks before marking the sections > - * onlining, such that undo_isolate_page_range() works correctly. > + * onlining, such that undo_isolate_pageblocks() works correctly. > */ > spin_lock_irqsave(&zone->lock, flags); > zone->nr_isolate_pageblock += nr_pages / pageblock_nr_pages; > @@ -1113,7 +1113,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, > build_all_zonelists(NULL); > > /* Basic onlining is complete, allow allocation of onlined pages. */ > - undo_isolate_page_range(pfn, pfn + nr_pages, MIGRATE_MOVABLE); > + undo_isolate_pageblocks(pfn, pfn + nr_pages, MIGRATE_MOVABLE); > > /* > * Freshly onlined pages aren't shuffled (e.g., all pages are placed to > @@ -1834,7 +1834,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, > lru_cache_disable(); > > /* set above range as isolated */ > - ret = start_isolate_page_range(start_pfn, end_pfn, > + ret = start_isolate_pageblocks(start_pfn, end_pfn, > MIGRATE_MOVABLE, > MEMORY_OFFLINE | REPORT_FAILURE); > if (ret) { > @@ -1937,7 +1937,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, > > failed_removal_isolated: > /* pushback to free area */ > - undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE); > + undo_isolate_pageblocks(start_pfn, end_pfn, MIGRATE_MOVABLE); > memory_notify(MEM_CANCEL_OFFLINE, &arg); > failed_removal_pcplists_disabled: > lru_cache_enable(); > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index badc05fc1b2f..76f8c19e37a2 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -8950,15 +8950,6 @@ void *__init alloc_large_system_hash(const char *tablename, > } > > #ifdef CONFIG_CONTIG_ALLOC > -static unsigned long pfn_max_align_down(unsigned long pfn) > -{ > - return ALIGN_DOWN(pfn, MAX_ORDER_NR_PAGES); > -} > - > -static unsigned long pfn_max_align_up(unsigned long pfn) > -{ > - return ALIGN(pfn, MAX_ORDER_NR_PAGES); > -} > > #if defined(CONFIG_DYNAMIC_DEBUG) || \ > (defined(CONFIG_DYNAMIC_DEBUG_CORE) && defined(DYNAMIC_DEBUG_MODULE)) > @@ -9104,8 +9095,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, > * put back to page allocator so that buddy can use them. > */ > > - ret = start_isolate_page_range(pfn_max_align_down(start), > - pfn_max_align_up(end), migratetype, 0); > + ret = start_isolate_page_range(start, end, migratetype, 0); > if (ret) > return ret; > > @@ -9186,8 +9176,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, > free_contig_range(end, outer_end - end); > > done: > - undo_isolate_page_range(pfn_max_align_down(start), > - pfn_max_align_up(end), migratetype); > + undo_isolate_page_range(start, end, migratetype); > return ret; > } > EXPORT_SYMBOL(alloc_contig_range); > diff --git a/mm/page_isolation.c b/mm/page_isolation.c > index b34f1310aeaa..7c1f7724c5bb 100644 > --- a/mm/page_isolation.c > +++ b/mm/page_isolation.c > @@ -15,6 +15,16 @@ > #define CREATE_TRACE_POINTS > #include > > +static unsigned long pfn_max_align_down(unsigned long pfn) > +{ > + return ALIGN_DOWN(pfn, MAX_ORDER_NR_PAGES); > +} > + > +static unsigned long pfn_max_align_up(unsigned long pfn) > +{ > + return ALIGN(pfn, MAX_ORDER_NR_PAGES); > +} > + > /* > * This function checks whether pageblock includes unmovable pages or not. > * > @@ -262,12 +272,40 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) > return NULL; > } > > +/* > + * Make page-allocation-type of pageblocks to be MIGRATE_ISOLATE. > + * > + * Most users should actually use start_isolate_page_range(). Only memory > + * onlining/offlining that knows exactly what it's doing in regard to > + * isolating only some pageblocks of MAX_ORDER - 1 pages (for the vmemmap) > + * should use this interface. > + */ > +int start_isolate_pageblocks(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype, int flags) > +{ > + unsigned long pfn; > + struct page *page; > + > + BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); > + BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); > + > + for (pfn = start_pfn; > + pfn < end_pfn; > + pfn += pageblock_nr_pages) { > + page = __first_valid_page(pfn, pageblock_nr_pages); > + if (page && set_migratetype_isolate(page, migratetype, flags)) { > + undo_isolate_pageblocks(start_pfn, pfn, migratetype); > + return -EBUSY; > + } > + } > + return 0; > +} > + > /** > * start_isolate_page_range() - make page-allocation-type of range of pages to > * be MIGRATE_ISOLATE. > * @start_pfn: The lower PFN of the range to be isolated. > * @end_pfn: The upper PFN of the range to be isolated. > - * start_pfn/end_pfn must be aligned to pageblock_order. > * @migratetype: Migrate type to set in error recovery. > * @flags: The following flags are allowed (they can be combined in > * a bit mask) > @@ -306,29 +344,23 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) > int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > unsigned migratetype, int flags) > { > - unsigned long pfn; > - struct page *page; > - > - BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); > - BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); > + start_pfn = = pfn_max_align_down(start_pfn); > + end_pfn = pfn_max_align_up(end_pfn); > > - for (pfn = start_pfn; > - pfn < end_pfn; > - pfn += pageblock_nr_pages) { > - page = __first_valid_page(pfn, pageblock_nr_pages); > - if (page && set_migratetype_isolate(page, migratetype, flags)) { > - undo_isolate_page_range(start_pfn, pfn, migratetype); > - return -EBUSY; > - } > - } > - return 0; > + return start_isolate_pageblocks(start_pfn, end_pfn, migratetype, flags); > } > > /* > - * Make isolated pages available again. > + * Make isolated pageblocks, isolated via start_isolate_pageblocks, available > + * again. > + * > + * Most users should actually use undo_isolate_page_range(). Only memory > + * onlining/offlining that knows exactly what it's doing in regard to > + * isolating only some pageblocks of MAX_ORDER - 1 pages (for the vmemmap) > + * should use this interface. > */ > -void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > - unsigned migratetype) > +void undo_isolate_pageblocks(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype) > { > unsigned long pfn; > struct page *page; > @@ -345,6 +377,21 @@ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > unset_migratetype_isolate(page, migratetype); > } > } > + > +/* > + * Make isolated pageblocks, isolated via start_isolate_page_range(), available > + * again. The pageblock isolation range will be extended just like for > + * start_isolate_page_range(). > + */ > +void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, > + unsigned migratetype) > +{ > + start_pfn = = pfn_max_align_down(start_pfn); > + end_pfn = pfn_max_align_up(end_pfn); > + > + return undo_isolate_pageblocks(start_pfn, end_pfn, migratetype); > +} > + > /* > * Test all pages in the range is free(means isolated) or not. > * all pages in [start_pfn...end_pfn) must be in the same zone. > -- > 2.35.1 > > > > -- > Thanks, > > David / dhildenb -- Best Regards, Yan, Zi