Re: Memory CG and 5.1 to 5.6 uprade slows backup

[Michal Hocko <mhocko@kernel.org>]

On Thu 09-04-20 12:17:33, Bruno Prémont wrote:
> On Thu, 9 Apr 2020 11:46:15 Michal Hocko <mhocko@kernel.org> wrote:
> > [Cc Chris]
> > 
> > On Thu 09-04-20 11:25:05, Bruno Prémont wrote:
> > > Hi,
> > > 
> > > Upgrading from 5.1 kernel to 5.6 kernel on a production system using
> > > cgroups (v2) and having backup process in a memory.high=2G cgroup
> > > sees backup being highly throttled (there are about 1.5T to be
> > > backuped).  
> > 
> > What does /proc/sys/vm/dirty_* say?
> 
> /proc/sys/vm/dirty_background_bytes:0
> /proc/sys/vm/dirty_background_ratio:10
> /proc/sys/vm/dirty_bytes:0
> /proc/sys/vm/dirty_expire_centisecs:3000
> /proc/sys/vm/dirty_ratio:20
> /proc/sys/vm/dirty_writeback_centisecs:500

Sorry, but I forgot ask for the total amount of memory. But it seems
this is 64GB and 10% dirty ration might mean a lot of dirty memory.
Does the same happen if you reduce those knobs to something smaller than
2G? _bytes alternatives should be useful for that purpose.

[...]

> > Is it possible that the reclaim is not making progress on too many
> > dirty pages and that triggers the back off mechanism that has been
> > implemented recently in  5.4 (have a look at 0e4b01df8659 ("mm,
> > memcg: throttle allocators when failing reclaim over memory.high")
> > and e26733e0d0ec ("mm, memcg: throttle allocators based on
> > ancestral memory.high").
> 
> Could be though in that case it's throttling the wrong task/cgroup
> as far as I can see (at least from cgroup's memory stats) or being
> blocked by state external to the cgroup.
> Will have a look at those patches so get a better idea at what they
> change.

Could you check where is the task of your interest throttled?
/proc/<pid>/stack should give you a clue.

-- 
Michal Hocko
SUSE Labs