Re: [RFC-PATCH 2/4] mm: Add __rcu_alloc_page_lockless() func.

["Paul E. McKenney" <paulmck@kernel.org>]

On Wed, Sep 23, 2020 at 11:37:06AM +0100, Mel Gorman wrote:
> On Tue, Sep 22, 2020 at 03:12:57PM +0200, Uladzislau Rezki wrote:
> > > > > > Yes, I do well remember that you are unhappy with this approach.
> > > > > > Unfortunately, thus far, there is no solution that makes all developers
> > > > > > happy.  You might be glad to hear that we are also looking into other
> > > > > > solutions, each of which makes some other developers unhappy.  So we
> > > > > > are at least not picking on you alone.  :-/
> > > > > 
> > > > > No worries I do not feel like a whipping boy here. But do expect me to
> > > > > argue against the approach. I would also appreciate it if there was some
> > > > > more information on other attempts, why they have failed. E.g. why
> > > > > pre-allocation is not an option that works well enough in most
> > > > > reasonable workloads.
> > > > Pre-allocating has some drawbacks:
> > > > 
> > > > a) It is impossible to predict how many pages will be required to
> > > >    cover a demand that is controlled by different workloads on
> > > >    various systems.
> > > 
> > > Yes, this is not trivial but not a rocket science either. Remember that
> > > you are relying on a very dumb watermark based pcp pool from the
> > > allocator.
> > >
> > We rely on it, indeed. If the pcp-cache is depleted our special work is
> > triggered to charge our local cache(few pages) such way will also initiate
> > the process of pre-featching pages from the buddy allocator populating
> > the depleted pcp-cache. I do not have any concern here.
> 
> It can interfere with ATOMIC allocations in critical paths in extreme
> circumstances as it potentially puts increased pressure on the emergency
> reserve as watermarks are bypassed. That adds to the risk of a functional
> failuure if reclaim fails to make progress.  The number of pages are likely
> to be limited and unpredictable. As it uses any PCP type, it potentially
> causes fragmention issues. For the last point, the allocations may be
> transient in the RCU case now but not guaranteed forever. As the API is
> in gfp.h, it's open to abuse so the next guy that comes along and thinks
> "I am critical no matter what the name says" will cause problems. While
> you could argue that would be caught in review, plenty of GFP flag abuses
> made it through review.
> 
> Fundamentally, this is simply shifting the problem from RCU to the page
> allocator because of the locking arrangements and hazard of acquiring zone
> lock is a raw spinlock is held on RT. It does not even make the timing
> predictable as an empty PCU list (for example, a full drain in low memory
> situations) may mean the emergency path is hit anyway. About all it changes
> is the timing of when the emergency path is hit in some circumstances --
> it's not fixing the problem, it's simply changing the shape.

All good points!

On the other hand, duplicating a portion of the allocator functionality
within RCU increases the amount of reserved memory, and needlessly most
of the time.

Is there some way that we can locklessly allocate memory, but return
failure instead of running down the emergency pool?  A change to the loop
that iterates over the migration types?  Or to the loop that iterates
over the zones?  Something else?

> > > Mimicing a similar implementation shouldn't be all that hard
> > > and you will get your own pool which doesn't affect other page allocator
> > > users as much as a bonus.
> > > 
> > I see your point Michal. As i mentioned before, it is important to avoid of
> > having such own pools, because the aim is not to waste memory resources. A
> > page will be returned back to "page allocator" as soon as a scheduler place  
> > our reclaim thread on a CPU and grace period is passed. So, the resource
> > can be used for other needs. What is important.
> 
> As the emergency path and synchronising can be hit no matter what, why
> not increase the pool temporarily after the emergency path is hit and
> shrink it again later if necessary?

If I understand what you are suggesting, this is in fact what Uladzislau's
prototyped commit 8c0a1269709d ("rcu/tree: Add a work to allocate
pages from regular context") on the -rcu "dev" branch is intended to do.
The issue, as Uladislau noted above, is that scheduler delays can prevent
these pool-increase actions until the point at which there is no memory.

> > Otherwise a memory footprint is increased what is bad for low memory
> > conditions when OOM is involved.
> 
> OOM would only be a major factor if the size of the pools meant the
> machine could not even operate or at least was severely degraded. However,
> depleting the PCPU lists for RCU may slow kswapd making reclaim progress
> and cause an OOM in itself, or at least an intervention by a userspace
> monitor that kills non-critical applications in the background when memory
> pressure exists.

When under emergency conditions, we have one page allocated per 500
objects passed to kvfree_rcu().  So the increase in total allocated
memory load due to this emergency path is quite small.

> > > > As for memory overhead, it is important to reduce it because of
> > > > embedded devices like phones, where a low memory condition is a
> > > > big issue. In that sense pre-allocating is something that we strongly
> > > > would like to avoid.
> > > 
> > > How big "machines" are we talking about here? I would expect that really
> > > tiny machines would have hard times to really fill up thousands of pages
> > > with pointers to free...
> > > 
> > I mentioned above. We can not rely on static model. We would like to
> > have a mechanism that gives back ASAP used pages to page allocator
> > for other needs.
> 
> After an emergency, temporarily increase the size of the pool to avoid
> hitting the emergency path again in the near future.

By which time we might well already be in OOM territory.  The emergency
situations can ramp up very quickly.

> > > Would a similar scaling as the page allocator feasible. Really I mostly
> > > do care about shared nature of the pcp allocator list that one user can
> > > easily monopolize with this API.
> > > 
> > I see your concern. pcplist can be monopolized by already existing API:
> > 
> >     while (i < 100)
> >         __get_free_page(GFP_NOWAIT | __GFP_NOWARN);
> 
> That's not the same class of abuse as it can go to the buddy lists to
> refill the correct PCP lists, avoid fragmentation issues, obeys watermarks
> and wakes kswapd if it's not awake already.

Good point, and we did try doing it this way.  Unfortunately, in current
!PREEMPT kernels, this approach can deadlock on one of the allocator
locks via call_rcu().  This experience caused us to look at lockless
allocator access.  It also inspired the unconditional PREEMPT_COUNT
approach, but that has its own detractors.  (Yes, we are of course still
persuing it as well.)

							Thanx, Paul