[PATCH] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[PATCH] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Baoquan He]

In kdump kernel, memcg usually is disabled with 'cgroup_disable=memory'
for saving memory. Now kdump kernel will always panic when dump vmcore
to local disk:

BUG: kernel NULL pointer dereference, address: 0000000000000ab8
PGD 5fcab067 P4D 5fcab067 PUD 5ff73067 PMD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 0 PID: 598 Comm: makedumpfile Not tainted 5.3.0+ #26
Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018
RIP: 0010:mem_cgroup_track_foreign_dirty_slowpath+0x38/0x140
Code: 55 48 8b 2d 6a bd 12 01 53 4c 8b 67 38 0f 1f 44 00 00 49 8b 06 48 89 e9 31 db be ff ff ff ff 48 8b 38 49 8d 84 24 d0 0a 00 00 <48> 39 78 e8 74 5a 48 8b 50 f8 48 39 ca 79 0e 44 8b 00 41 83 f8 01
RSP: 0018:ffffbc9300817bd0 EFLAGS: 00010046
RAX: 0000000000000ad0 RBX: 0000000000000000 RCX: 00000000fffba227
RDX: fffffffffffffff8 RSI: 00000000ffffffff RDI: 0000000000000004
RBP: 00000000fffba227 R08: 0000000000030340 R09: ffffbc9300817d10
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000
R13: ffff999c1fa2bcf0 R14: ffff999c1f7bcc78 R15: ffffe51fc177a8c0
FS:  00007f84f14c2b80(0000) GS:ffff999c22200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000ab8 CR3: 000000005f564000 CR4: 00000000003406b0
Call Trace:
 __set_page_dirty+0x52/0xc0
 iomap_set_page_dirty+0x50/0x90
 iomap_write_end+0x6e/0x270
 iomap_write_actor+0xce/0x170
 ? iomap_write_end+0x270/0x270
 iomap_apply+0xba/0x11e
 ? iomap_write_end+0x270/0x270
 iomap_file_buffered_write+0x62/0x90
 ? iomap_write_end+0x270/0x270
 xfs_file_buffered_aio_write+0xca/0x320 [xfs]
 new_sync_write+0x12d/0x1d0
 vfs_write+0xa5/0x1a0
 ksys_write+0x59/0xd0
 do_syscall_64+0x59/0x1e0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f84f1093ab5

And this will corrupt the 1st kernel too with 'cgroup_disable=memory'.

From the trace and with debugging, it is pointing to commit 97b27821b485
("writeback, memcg: Implement foreign dirty flushing") which introduced
this regression. Disabling memcg causes the null pointer dereference at
uninitialized data in function mem_cgroup_track_foreign_dirty_slowpath().

Fix it by returning directly if memcg is disabled, but not trying to
record the foreign writebacks with dirty pages.

Fixed: 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing")
Signed-off-by: Baoquan He <bhe@redhat.com>
---
 include/linux/memcontrol.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index ad8f1a397ae4..fa53f9d51205 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
 static inline void mem_cgroup_track_foreign_dirty(struct page *page,
 						  struct bdi_writeback *wb)
 {
-	if (unlikely(&page->mem_cgroup->css != wb->memcg_css))
+	if (unlikely(&page->mem_cgroup->css != wb->memcg_css)
+		&& !mem_cgroup_disabled())
 		mem_cgroup_track_foreign_dirty_slowpath(page, wb);
 }
 
-- 
2.17.2

Re: [PATCH] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Baoquan He]

On 09/23/19 at 04:30pm, Baoquan He wrote:
 
> ---
>  include/linux/memcontrol.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index ad8f1a397ae4..fa53f9d51205 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
>  static inline void mem_cgroup_track_foreign_dirty(struct page *page,
>  						  struct bdi_writeback *wb)
>  {
> -	if (unlikely(&page->mem_cgroup->css != wb->memcg_css))
> +	if (unlikely(&page->mem_cgroup->css != wb->memcg_css)
> +		&& !mem_cgroup_disabled())

Sorry, this is the draft patch I was testing. Later I think this
had better be moved into mem_cgroup_track_foreign_dirty_slowpath().

Not very sure about this. Will send a v2 to match the patch log.

>  		mem_cgroup_track_foreign_dirty_slowpath(page, wb);
>  }
>  
> -- 
> 2.17.2
> 

[PATCH v2] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Baoquan He]

In kdump kernel, memcg usually is disabled with 'cgroup_disable=memory'
for saving memory. Now kdump kernel will always panic when dump vmcore
to local disk:

BUG: kernel NULL pointer dereference, address: 0000000000000ab8
PGD 5fcab067 P4D 5fcab067 PUD 5ff73067 PMD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 0 PID: 598 Comm: makedumpfile Not tainted 5.3.0+ #26
Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018
RIP: 0010:mem_cgroup_track_foreign_dirty_slowpath+0x38/0x140
Code: 55 48 8b 2d 6a bd 12 01 53 4c 8b 67 38 0f 1f 44 00 00 49 8b 06 48 89 e9 31 db be ff ff ff ff 48 8b 38 49 8d 84 24 d0 0a 00 00 <48> 39 78 e8
74 5a 48 8b 50 f8 48 39 ca 79 0e 44 8b 00 41 83 f8 01
RSP: 0018:ffffbc9300817bd0 EFLAGS: 00010046
RAX: 0000000000000ad0 RBX: 0000000000000000 RCX: 00000000fffba227
RDX: fffffffffffffff8 RSI: 00000000ffffffff RDI: 0000000000000004
RBP: 00000000fffba227 R08: 0000000000030340 R09: ffffbc9300817d10
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000
R13: ffff999c1fa2bcf0 R14: ffff999c1f7bcc78 R15: ffffe51fc177a8c0
FS:  00007f84f14c2b80(0000) GS:ffff999c22200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000ab8 CR3: 000000005f564000 CR4: 00000000003406b0
Call Trace:
 __set_page_dirty+0x52/0xc0
 iomap_set_page_dirty+0x50/0x90
 iomap_write_end+0x6e/0x270
 iomap_write_actor+0xce/0x170
 ? iomap_write_end+0x270/0x270
 iomap_apply+0xba/0x11e
 ? iomap_write_end+0x270/0x270
 iomap_file_buffered_write+0x62/0x90
 ? iomap_write_end+0x270/0x270
 xfs_file_buffered_aio_write+0xca/0x320 [xfs]
 new_sync_write+0x12d/0x1d0
 vfs_write+0xa5/0x1a0
 ksys_write+0x59/0xd0
 do_syscall_64+0x59/0x1e0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f84f1093ab5

And this will corrupt the 1st kernel too with 'cgroup_disable=memory'.

From the trace and with debugging, it is pointing to commit 97b27821b485
("writeback, memcg: Implement foreign dirty flushing") which introduced
this regression. Disabling memcg causes the null pointer dereference at
uninitialized data in function mem_cgroup_track_foreign_dirty_slowpath().

Fix it by returning directly if memcg is disabled, but not trying to
record the foreign writebacks with dirty pages.

Fixed: 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing")
Signed-off-by: Baoquan He <bhe@redhat.com>
---
v1->v2:
  Move the checking code into mem_cgroup_track_foreign_dirty_slowpath()
  because usually memcg is enabled by default, and the original condition
  checking in mem_cgroup_track_foreign_dirty() is more unlikely being
  true to make it return.

 mm/memcontrol.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f3c15bb07cce..84e3fdb1ccb4 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
 
 	trace_track_foreign_dirty(page, wb);
 
+	if (mem_cgroup_disabled())
+		return;
+
 	/*
 	 * Pick the slot to use.  If there is already a slot for @wb, keep
 	 * using it.  If not replace the oldest one which isn't being
-- 
2.17.2

Re: [PATCH v2] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Michal Hocko]

On Tue 24-09-19 19:11:51, Baoquan He wrote:
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index f3c15bb07cce..84e3fdb1ccb4 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
>  
>  	trace_track_foreign_dirty(page, wb);
>  
> +	if (mem_cgroup_disabled())
> +		return;
> +

This doesn't seem correct. We shouldn't even enter the slowpath with
memcg disabled AFAIC. The check should be done at mem_cgroup_track_foreign_dirty
level.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH v2] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Baoquan He]

On 09/24/19 at 02:27pm, Michal Hocko wrote:
> On Tue 24-09-19 19:11:51, Baoquan He wrote:
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index f3c15bb07cce..84e3fdb1ccb4 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
> >  
> >  	trace_track_foreign_dirty(page, wb);
> >  
> > +	if (mem_cgroup_disabled())
> > +		return;
> > +
> 
> This doesn't seem correct. We shouldn't even enter the slowpath with
> memcg disabled AFAIC. The check should be done at mem_cgroup_track_foreign_dirty
> level.

You mean the way in v1 patch, right? It's also fine to me.

I am worried about the case that memcg is enabled, the checking by
calling mem_cgroup_disabled() will lower efficiency. And it entering
into mem_cgroup_track_foreign_dirty_slowpath() should be a rare event.

Thanks
Baoquan

Re: [PATCH v2] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Michal Hocko]

On Tue 24-09-19 21:04:58, Baoquan He wrote:
> On 09/24/19 at 02:27pm, Michal Hocko wrote:
> > On Tue 24-09-19 19:11:51, Baoquan He wrote:
> > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > index f3c15bb07cce..84e3fdb1ccb4 100644
> > > --- a/mm/memcontrol.c
> > > +++ b/mm/memcontrol.c
> > > @@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
> > >  
> > >  	trace_track_foreign_dirty(page, wb);
> > >  
> > > +	if (mem_cgroup_disabled())
> > > +		return;
> > > +
> > 
> > This doesn't seem correct. We shouldn't even enter the slowpath with
> > memcg disabled AFAIC. The check should be done at mem_cgroup_track_foreign_dirty
> > level.
> 
> You mean the way in v1 patch, right? It's also fine to me.
> 
> I am worried about the case that memcg is enabled, the checking by
> calling mem_cgroup_disabled() will lower efficiency.

This is hidden by a static branch so I wouldn't really be worried about
the overhead.

> And it entering
> into mem_cgroup_track_foreign_dirty_slowpath() should be a rare event.

But &page->mem_cgroup->css != wb->memcg_css doesn't make any sense when
memcg is disabled, right?
-- 
Michal Hocko
SUSE Labs

Re: [PATCH v2] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Baoquan He]

On 09/24/19 at 03:16pm, Michal Hocko wrote:
> On Tue 24-09-19 21:04:58, Baoquan He wrote:
> > On 09/24/19 at 02:27pm, Michal Hocko wrote:
> > > On Tue 24-09-19 19:11:51, Baoquan He wrote:
> > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > > index f3c15bb07cce..84e3fdb1ccb4 100644
> > > > --- a/mm/memcontrol.c
> > > > +++ b/mm/memcontrol.c
> > > > @@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
> > > >  
> > > >  	trace_track_foreign_dirty(page, wb);
> > > >  
> > > > +	if (mem_cgroup_disabled())
> > > > +		return;
> > > > +
> > > 
> > > This doesn't seem correct. We shouldn't even enter the slowpath with
> > > memcg disabled AFAIC. The check should be done at mem_cgroup_track_foreign_dirty
> > > level.
> > 
> > You mean the way in v1 patch, right? It's also fine to me.
> > 
> > I am worried about the case that memcg is enabled, the checking by
> > calling mem_cgroup_disabled() will lower efficiency.
> 
> This is hidden by a static branch so I wouldn't really be worried about
> the overhead.
> 
> > And it entering
> > into mem_cgroup_track_foreign_dirty_slowpath() should be a rare event.
> 
> But &page->mem_cgroup->css != wb->memcg_css doesn't make any sense when
> memcg is disabled, right?

Yeah, I think so. Make it like below?

@@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
 static inline void mem_cgroup_track_foreign_dirty(struct page *page,
                                                  struct bdi_writeback *wb)
 {
-       if (unlikely(&page->mem_cgroup->css != wb->memcg_css))
+       if (!mem_cgroup_disabled() &&
+           unlikely(&page->mem_cgroup->css != wb->memcg_css))
                mem_cgroup_track_foreign_dirty_slowpath(page, wb);
 }

Re: [PATCH v2] memcg: Only record foreign writebacks with dirty pages when memcg is not disabled

[Michal Hocko]

On Tue 24-09-19 21:43:52, Baoquan He wrote:
> On 09/24/19 at 03:16pm, Michal Hocko wrote:
> > On Tue 24-09-19 21:04:58, Baoquan He wrote:
> > > On 09/24/19 at 02:27pm, Michal Hocko wrote:
> > > > On Tue 24-09-19 19:11:51, Baoquan He wrote:
> > > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > > > index f3c15bb07cce..84e3fdb1ccb4 100644
> > > > > --- a/mm/memcontrol.c
> > > > > +++ b/mm/memcontrol.c
> > > > > @@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
> > > > >  
> > > > >  	trace_track_foreign_dirty(page, wb);
> > > > >  
> > > > > +	if (mem_cgroup_disabled())
> > > > > +		return;
> > > > > +
> > > > 
> > > > This doesn't seem correct. We shouldn't even enter the slowpath with
> > > > memcg disabled AFAIC. The check should be done at mem_cgroup_track_foreign_dirty
> > > > level.
> > > 
> > > You mean the way in v1 patch, right? It's also fine to me.
> > > 
> > > I am worried about the case that memcg is enabled, the checking by
> > > calling mem_cgroup_disabled() will lower efficiency.
> > 
> > This is hidden by a static branch so I wouldn't really be worried about
> > the overhead.
> > 
> > > And it entering
> > > into mem_cgroup_track_foreign_dirty_slowpath() should be a rare event.
> > 
> > But &page->mem_cgroup->css != wb->memcg_css doesn't make any sense when
> > memcg is disabled, right?
> 
> Yeah, I think so. Make it like below?

Or just put it on its own line to make the code more readable.

> @@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
>  static inline void mem_cgroup_track_foreign_dirty(struct page *page,
>                                                   struct bdi_writeback *wb)
>  {
> -       if (unlikely(&page->mem_cgroup->css != wb->memcg_css))
> +       if (!mem_cgroup_disabled() &&
> +           unlikely(&page->mem_cgroup->css != wb->memcg_css))
>                 mem_cgroup_track_foreign_dirty_slowpath(page, wb);
>  }

-- 
Michal Hocko
SUSE Labs